Hi Lukas,
My reply to your question was stuck in draft.. :( Pasting it below just for
reference. You are right. I guess your option is to create a symlink.
*Thanks for clarifying that you are not using Yarn. *
*In Samza container implementation, we don't seem to allow any
configuration related to the directory path for the state stores. Not sure
about the story behind this. But by default, the path will be set to
System.getenv("user.dir")/state/${TaskName}/*
*Check this line
->
https://github.com/apache/samza/blob/0.9.0/samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala#L374
<https://github.com/apache/samza/blob/0.9.0/samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala#L374>*
*To answer your question, there is no configuration, as of today, that
allows you to specify the directory path for the state stores.*
Regards,
Navina
On Tue, Nov 10, 2015 at 3:33 AM, Lukáš Havrlant <lu...@havrlant.cz> wrote:
> Hi,
> I managed to start Samza after all. I am not sure what has helped
> eventually. But still I had to create a folder /opt/samza/state (/opt/samza
> is Samza's Home folder) and I don't want to have this folder there. Is
> there any option for it? I looked at the configuration page (
>
> http://samza.apache.org/learn/documentation/0.9/jobs/configuration-table.html
> )
> and I don't see such an option. It's not an emergency because I can always
> create symlink, but still...
>
> Lukáš Havrlant
>
> 2015-11-09 23:28 GMT+01:00 Lukáš Havrlant <lu...@havrlant.cz>:
>
> > Hi Navina,
> > I am sorry I didn't mentioned it before. We run Samza as a local job:
> > job.factory.class=org.apache.samza.job.local.ThreadJobFactory But thank
> for
> > your answer regarding YARN, I'll be setting up Samza on YARN next week!
> >
> > If I remove all the ROCKSDB_SHAREDLIB_DIR stuff I get this error:
> >
> > Exception in thread "main" java.lang.ExceptionInInitializerError
> > at org.rocksdb.Options.<clinit>(Options.java:17)
> > at
> >
> org.apache.samza.storage.kv.RocksDbKeyValueStore$.options(RocksDbKeyValueStore.scala:32)
> > at
> >
> org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory.getKVStore(RocksDbKeyValueStorageEngineFactory.scala:47)
> > ...
> > at org.apache.samza.job.JobRunner.main(JobRunner.scala)
> > Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared
> > libraryjava.io.IOException: Permission Denied
> > at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:54)
> > at org.rocksdb.RocksDB.<clinit>(RocksDB.java:28)
> > ... 27 more
> >
> > It should be this line in code:
> >
> https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/RocksDB.java#L51
> >
> > Is there a way how to change the output directory for ThreadJobFactory? I
> > guess Samza wants to extract the .so libraries somewhere into /opt/samza
> > (the "home dir" for our Samza) and the process does not have the
> privilege
> > to do so.
> >
> > Lukáš
> >
> > 2015-11-09 20:12 GMT+01:00 Navina Ramesh <nram...@linkedin.com>:
> >
> >> Hi Lukas,
> >>
> >> Are you running the Samza job on Yarn? If so, the state stores can be
> >> found
> >> in the working directory of a yarn application. Samza doesn't configure
> >> any
> >> special location. Unless you have configured the local-dir location in
> >> yarn, it should use the default, which is
> >>
> >>
> *${hadoop.tmp.dir}/nm-local-dir/usercache/${user}/appcache/application_${appid}.
> >> *
> >> Within each application directory, you can find the stores in
> >> *state/${store-name}/${taskName}/* .
> >>
> >> Please note that if yarn.nodemanager.delete.debug-delay-sec is not set >
> >> 0,
> >> the working directory gets deleted once the job fails. Hence, you won't
> be
> >> able to find any logs or state directory.
> >>
> >> HTH. It will be great if you can provide the stack trace so that we can
> >> narrow down the scope of your problem further.
> >>
> >> Thanks!
> >> Navina
> >>
> >> On Mon, Nov 9, 2015 at 9:22 AM, Lukáš Havrlant <lu...@havrlant.cz>
> wrote:
> >>
> >> > Yes I think I saw something like "permission denied" in the logs. But
> I
> >> > cannot find the config that tells the Samza the destination folder. We
> >> > tried to set different -Djava.io.tmpdir but it didn't help. Can you
> >> please
> >> > give me a hint what config should it be?
> >> >
> >> > 2015-11-09 18:05 GMT+01:00 Tommy Becker <tobec...@tivo.com>:
> >> >
> >> > > The native RocksDB libs are included in the jar file, but they are
> >> > > extracted at runtime before they can be loaded. Make sure the user
> >> > running
> >> > > the job has permission to write to...wherever it is that it tries to
> >> put
> >> > it
> >> > > (sorry don't know offhand). But you shouldn't have to set any
> >> environment
> >> > > variables or anything for it to work.
> >> > >
> >> > >
> >> > > On 11/09/2015 08:50 AM, Lukáš Havrlant wrote:
> >> > >
> >> > > Hello,
> >> > > we have a problem with configuring RocksDB as a key-value store with
> >> our
> >> > > Samza 0.9.0 project. We added this line to our config:
> >> > >
> >> > >
> >> > >
> >> >
> >>
> stores.my-store.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory
> >> > >
> >> > > But then our Samza job couldn't locate the
> >> > > RocksDbKeyValueStorageEngineFactory class. So we added compile
> >> > > "org.apache.samza:samza-kv-rocksdb_2.10:$samzaVersion" to our gradle
> >> > build
> >> > > script. It helped but now Samza failed to start because it couldn't
> >> > > find librocksdbjni-linux64.so library. We added export
> >> > > ROCKSDB_SHAREDLIB_DIR=/opt/samza/lib to our init script and it
> didn't
> >> > help.
> >> > > Then we manually copied the librocksdbjni-linux64.so from
> >> > > the rocksdbjni-3.5.1.jar to the /opt/samza/lib folder and Samza
> still
> >> > > couldn't start, now because of
> >> > >
> >> > > 2015-11-09T14:35:06,209 INFO [main]
> >> > > org.apache.samza.container.SamzaContainer$ - Got store consumers:
> >> > > Map(tracking-info ->
> >> > > org.apache.samza.system.kafka.KafkaSystemConsumer@78461bc4)
> >> > > # A fatal error has been detected by the Java Runtime Environment:
> >> > > #
> >> > > # SIGSEGV (0xb) at pc=0x0000003c4de092ab, pid=4410,
> >> tid=139728971949824
> >> > >
> >> > > And we're out of ideas at the moment. Can you please help me set up
> >> > RocksDB
> >> > > correctly? I suppose we are doing something completely wrong.
> >> > >
> >> > > Is it possibble to use another key-value store at least?
> >> > >
> >> > > Lukáš Havrlant
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Tommy Becker
> >> > > Senior Software Engineer
> >> > >
> >> > > Digitalsmiths
> >> > > A TiVo Company
> >> > >
> >> > > www.digitalsmiths.com<http://www.digitalsmiths.com>
> >> > > tobec...@tivo.com<mailto:tobec...@tivo.com>
> >> > >
> >> > > ________________________________
> >> > >
> >> > > This email and any attachments may contain confidential and
> privileged
> >> > > material for the sole use of the intended recipient. Any review,
> >> copying,
> >> > > or distribution of this email (or any attachments) by others is
> >> > prohibited.
> >> > > If you are not the intended recipient, please contact the sender
> >> > > immediately and permanently delete this email and any attachments.
> No
> >> > > employee or agent of TiVo Inc. is authorized to conclude any binding
> >> > > agreement on behalf of TiVo Inc. by email. Binding agreements with
> >> TiVo
> >> > > Inc. may only be made by a signed written agreement.
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Navina R.
> >>
> >
> >
>
--
Navina R.
