Hi, I managed to start Samza after all. I am not sure what has helped eventually. But still I had to create a folder /opt/samza/state (/opt/samza is Samza's Home folder) and I don't want to have this folder there. Is there any option for it? I looked at the configuration page ( http://samza.apache.org/learn/documentation/0.9/jobs/configuration-table.html) and I don't see such an option. It's not an emergency because I can always create symlink, but still...
Lukáš Havrlant 2015-11-09 23:28 GMT+01:00 Lukáš Havrlant <lu...@havrlant.cz>: > Hi Navina, > I am sorry I didn't mentioned it before. We run Samza as a local job: > job.factory.class=org.apache.samza.job.local.ThreadJobFactory But thank for > your answer regarding YARN, I'll be setting up Samza on YARN next week! > > If I remove all the ROCKSDB_SHAREDLIB_DIR stuff I get this error: > > Exception in thread "main" java.lang.ExceptionInInitializerError > at org.rocksdb.Options.<clinit>(Options.java:17) > at > org.apache.samza.storage.kv.RocksDbKeyValueStore$.options(RocksDbKeyValueStore.scala:32) > at > org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory.getKVStore(RocksDbKeyValueStorageEngineFactory.scala:47) > ... > at org.apache.samza.job.JobRunner.main(JobRunner.scala) > Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared > libraryjava.io.IOException: Permission Denied > at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:54) > at org.rocksdb.RocksDB.<clinit>(RocksDB.java:28) > ... 27 more > > It should be this line in code: > https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/RocksDB.java#L51 > > Is there a way how to change the output directory for ThreadJobFactory? I > guess Samza wants to extract the .so libraries somewhere into /opt/samza > (the "home dir" for our Samza) and the process does not have the privilege > to do so. > > Lukáš > > 2015-11-09 20:12 GMT+01:00 Navina Ramesh <nram...@linkedin.com>: > >> Hi Lukas, >> >> Are you running the Samza job on Yarn? If so, the state stores can be >> found >> in the working directory of a yarn application. Samza doesn't configure >> any >> special location. Unless you have configured the local-dir location in >> yarn, it should use the default, which is >> >> *${hadoop.tmp.dir}/nm-local-dir/usercache/${user}/appcache/application_${appid}. >> * >> Within each application directory, you can find the stores in >> *state/${store-name}/${taskName}/* . >> >> Please note that if yarn.nodemanager.delete.debug-delay-sec is not set > >> 0, >> the working directory gets deleted once the job fails. Hence, you won't be >> able to find any logs or state directory. >> >> HTH. It will be great if you can provide the stack trace so that we can >> narrow down the scope of your problem further. >> >> Thanks! >> Navina >> >> On Mon, Nov 9, 2015 at 9:22 AM, Lukáš Havrlant <lu...@havrlant.cz> wrote: >> >> > Yes I think I saw something like "permission denied" in the logs. But I >> > cannot find the config that tells the Samza the destination folder. We >> > tried to set different -Djava.io.tmpdir but it didn't help. Can you >> please >> > give me a hint what config should it be? >> > >> > 2015-11-09 18:05 GMT+01:00 Tommy Becker <tobec...@tivo.com>: >> > >> > > The native RocksDB libs are included in the jar file, but they are >> > > extracted at runtime before they can be loaded. Make sure the user >> > running >> > > the job has permission to write to...wherever it is that it tries to >> put >> > it >> > > (sorry don't know offhand). But you shouldn't have to set any >> environment >> > > variables or anything for it to work. >> > > >> > > >> > > On 11/09/2015 08:50 AM, Lukáš Havrlant wrote: >> > > >> > > Hello, >> > > we have a problem with configuring RocksDB as a key-value store with >> our >> > > Samza 0.9.0 project. We added this line to our config: >> > > >> > > >> > > >> > >> stores.my-store.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory >> > > >> > > But then our Samza job couldn't locate the >> > > RocksDbKeyValueStorageEngineFactory class. So we added compile >> > > "org.apache.samza:samza-kv-rocksdb_2.10:$samzaVersion" to our gradle >> > build >> > > script. It helped but now Samza failed to start because it couldn't >> > > find librocksdbjni-linux64.so library. We added export >> > > ROCKSDB_SHAREDLIB_DIR=/opt/samza/lib to our init script and it didn't >> > help. >> > > Then we manually copied the librocksdbjni-linux64.so from >> > > the rocksdbjni-3.5.1.jar to the /opt/samza/lib folder and Samza still >> > > couldn't start, now because of >> > > >> > > 2015-11-09T14:35:06,209 INFO [main] >> > > org.apache.samza.container.SamzaContainer$ - Got store consumers: >> > > Map(tracking-info -> >> > > org.apache.samza.system.kafka.KafkaSystemConsumer@78461bc4) >> > > # A fatal error has been detected by the Java Runtime Environment: >> > > # >> > > # SIGSEGV (0xb) at pc=0x0000003c4de092ab, pid=4410, >> tid=139728971949824 >> > > >> > > And we're out of ideas at the moment. Can you please help me set up >> > RocksDB >> > > correctly? I suppose we are doing something completely wrong. >> > > >> > > Is it possibble to use another key-value store at least? >> > > >> > > Lukáš Havrlant >> > > >> > > >> > > >> > > -- >> > > Tommy Becker >> > > Senior Software Engineer >> > > >> > > Digitalsmiths >> > > A TiVo Company >> > > >> > > www.digitalsmiths.com<http://www.digitalsmiths.com> >> > > tobec...@tivo.com<mailto:tobec...@tivo.com> >> > > >> > > ________________________________ >> > > >> > > This email and any attachments may contain confidential and privileged >> > > material for the sole use of the intended recipient. Any review, >> copying, >> > > or distribution of this email (or any attachments) by others is >> > prohibited. >> > > If you are not the intended recipient, please contact the sender >> > > immediately and permanently delete this email and any attachments. No >> > > employee or agent of TiVo Inc. is authorized to conclude any binding >> > > agreement on behalf of TiVo Inc. by email. Binding agreements with >> TiVo >> > > Inc. may only be made by a signed written agreement. >> > > >> > >> >> >> >> -- >> Navina R. >> > >
