Hi Lukas,
Are you running the Samza job on Yarn? If so, the state stores can be found
in the working directory of a yarn application. Samza doesn't configure any
special location. Unless you have configured the local-dir location in
yarn, it should use the default, which is
*${hadoop.tmp.dir}/nm-local-dir/usercache/${user}/appcache/application_${appid}.
*
Within each application directory, you can find the stores in
*state/${store-name}/${taskName}/* .
Please note that if yarn.nodemanager.delete.debug-delay-sec is not set > 0,
the working directory gets deleted once the job fails. Hence, you won't be
able to find any logs or state directory.
HTH. It will be great if you can provide the stack trace so that we can
narrow down the scope of your problem further.
Thanks!
Navina
On Mon, Nov 9, 2015 at 9:22 AM, Lukáš Havrlant <lu...@havrlant.cz> wrote:
> Yes I think I saw something like "permission denied" in the logs. But I
> cannot find the config that tells the Samza the destination folder. We
> tried to set different -Djava.io.tmpdir but it didn't help. Can you please
> give me a hint what config should it be?
>
> 2015-11-09 18:05 GMT+01:00 Tommy Becker <tobec...@tivo.com>:
>
> > The native RocksDB libs are included in the jar file, but they are
> > extracted at runtime before they can be loaded. Make sure the user
> running
> > the job has permission to write to...wherever it is that it tries to put
> it
> > (sorry don't know offhand). But you shouldn't have to set any environment
> > variables or anything for it to work.
> >
> >
> > On 11/09/2015 08:50 AM, Lukáš Havrlant wrote:
> >
> > Hello,
> > we have a problem with configuring RocksDB as a key-value store with our
> > Samza 0.9.0 project. We added this line to our config:
> >
> >
> >
> stores.my-store.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory
> >
> > But then our Samza job couldn't locate the
> > RocksDbKeyValueStorageEngineFactory class. So we added compile
> > "org.apache.samza:samza-kv-rocksdb_2.10:$samzaVersion" to our gradle
> build
> > script. It helped but now Samza failed to start because it couldn't
> > find librocksdbjni-linux64.so library. We added export
> > ROCKSDB_SHAREDLIB_DIR=/opt/samza/lib to our init script and it didn't
> help.
> > Then we manually copied the librocksdbjni-linux64.so from
> > the rocksdbjni-3.5.1.jar to the /opt/samza/lib folder and Samza still
> > couldn't start, now because of
> >
> > 2015-11-09T14:35:06,209 INFO [main]
> > org.apache.samza.container.SamzaContainer$ - Got store consumers:
> > Map(tracking-info ->
> > org.apache.samza.system.kafka.KafkaSystemConsumer@78461bc4)
> > # A fatal error has been detected by the Java Runtime Environment:
> > #
> > # SIGSEGV (0xb) at pc=0x0000003c4de092ab, pid=4410, tid=139728971949824
> >
> > And we're out of ideas at the moment. Can you please help me set up
> RocksDB
> > correctly? I suppose we are doing something completely wrong.
> >
> > Is it possibble to use another key-value store at least?
> >
> > Lukáš Havrlant
> >
> >
> >
> > --
> > Tommy Becker
> > Senior Software Engineer
> >
> > Digitalsmiths
> > A TiVo Company
> >
> > www.digitalsmiths.com<http://www.digitalsmiths.com>
> > tobec...@tivo.com<mailto:tobec...@tivo.com>
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>
--
Navina R.
