Storm version is 0.9.5
So I have been moving along with the storm stuff and I have decided to give
“–config” a try. I need to know if this is a bug or I am doing something wrong.
And it takes the –config part just fine as an argument however the java part is
puking on it and failing, see below:
ls -ald /etc/storm/conf
lrwxrwxrwx 1 root root 28 Oct 27 14:32 /etc/storm/conf ->
/etc/alternatives/storm-conf
ls -al /etc/alternatives/storm-conf
lrwxrwxrwx 1 root root 41 Oct 27 14:32 /etc/alternatives/storm-conf ->
/opt/cloudera/parcels/STORM-0.9.5-u1/conf
ls -ald /opt/cloudera/parcels/STORM-0.9.5-u1/conf
drwxrwxr-x 2 storm storm 4096 Nov 2 12:01
/opt/cloudera/parcels/STORM-0.9.5-u1/conf
ls -al /opt/cloudera/parcels/STORM-0.9.5-u1/conf
total 16
drwxrwxr-x 2 storm storm 4096 Nov 2 12:01 .
drwxrwxr-x 11 root root 4096 Oct 26 12:00 ..
-rwxrwxr-x 1 root root 1128 Sep 17 14:48 storm_env.ini
-rwxrwxr-x 1 storm storm 422 Nov 2 11:42 storm_nimbus.yaml
ls -al /etc/storm/conf/
total 16
drwxrwxr-x 2 storm storm 4096 Nov 2 12:01 .
drwxrwxr-x 11 root root 4096 Oct 26 12:00 ..
-rwxrwxr-x 1 root root 1128 Sep 17 14:48 storm_env.ini
-rwxrwxr-x 1 storm storm 422 Nov 2 11:42 storm_nimbus.yaml
storm nimbus --config /etc/storm/conf/storm_nimbus.yaml
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:195)
at
backtype.storm.config$loading__4910__auto__.invoke(config.clj:17)
at backtype.storm.config__init.load(Unknown Source)
at backtype.storm.config__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at clojure.lang.RT.loadClassForName(RT.java:2098)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5336)
at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
at clojure.core$load_lib.doInvoke(core.clj:5374)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:619)
at clojure.core$load_libs.doInvoke(core.clj:5417)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:621)
at clojure.core$use.doInvoke(core.clj:5507)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at
backtype.storm.command.config_value$loading__4910__auto__.invoke(config_value.clj:16)
at backtype.storm.command.config_value__init.load(Unknown Source)
at backtype.storm.command.config_value__init.<clinit>(Unknown
Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at clojure.lang.RT.loadClassForName(RT.java:2098)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:415)
at backtype.storm.command.config_value.<clinit>(Unknown Source)
Caused by: java.lang.RuntimeException: Could not find config file on classpath
/etc/storm/conf/storm_nimbus.yaml
at
backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:190)
at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
... 36 more
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:195)
at
backtype.storm.config$loading__4910__auto__.invoke(config.clj:17)
at backtype.storm.config__init.load(Unknown Source)
at backtype.storm.config__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at clojure.lang.RT.loadClassForName(RT.java:2098)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5336)
at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
at clojure.core$load_lib.doInvoke(core.clj:5374)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:619)
at clojure.core$load_libs.doInvoke(core.clj:5417)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:621)
at clojure.core$use.doInvoke(core.clj:5507)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at
backtype.storm.command.config_value$loading__4910__auto__.invoke(config_value.clj:16)
at backtype.storm.command.config_value__init.load(Unknown Source)
at backtype.storm.command.config_value__init.<clinit>(Unknown
Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at clojure.lang.RT.loadClassForName(RT.java:2098)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:415)
at backtype.storm.command.config_value.<clinit>(Unknown Source)
Caused by: java.lang.RuntimeException: Could not find config file on classpath
/etc/storm/conf/storm_nimbus.yaml
at
backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:190)
at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
... 36 more
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:195)
at
backtype.storm.config$loading__4910__auto__.invoke(config.clj:17)
at backtype.storm.config__init.load(Unknown Source)
at backtype.storm.config__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at clojure.lang.RT.loadClassForName(RT.java:2098)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5336)
at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
at clojure.core$load_lib.doInvoke(core.clj:5374)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:619)
at clojure.core$load_libs.doInvoke(core.clj:5417)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:621)
at clojure.core$use.doInvoke(core.clj:5507)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at
backtype.storm.command.config_value$loading__4910__auto__.invoke(config_value.clj:16)
at backtype.storm.command.config_value__init.load(Unknown Source)
at backtype.storm.command.config_value__init.<clinit>(Unknown
Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at clojure.lang.RT.loadClassForName(RT.java:2098)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:415)
at backtype.storm.command.config_value.<clinit>(Unknown Source)
Caused by: java.lang.RuntimeException: Could not find config file on classpath
/etc/storm/conf/storm_nimbus.yaml
at
backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:190)
at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
... 36 more
Running: java -server -Dstorm.options=
-Dstorm.home=/opt/cloudera/parcels/STORM-0.9.5-u1 -Dstorm.log.dir=
-Djava.library.path= -Dstorm.conf.file=/etc/storm/conf/storm_nimbus.yaml -cp
/opt/cloudera/parcels/STORM-0.9.5-u1/lib/jgrapht-core-0.9.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/tools.cli-0.2.4.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-codec-1.6.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/jline-2.11.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/asm-4.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/tools.macro-0.1.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/math.numeric-tower-0.0.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/joda-time-2.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/jetty-util-6.1.26.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/disruptor-2.10.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/objenesis-1.2.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-io-2.4.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/clout-1.0.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/clojure-1.5.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/chill-java-0.3.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/ring-servlet-0.3.11.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/core.incubator-0.1.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/ring-core-1.1.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/storm-core-0.9.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/log4j-over-slf4j-1.6.6.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/compojure-1.1.3.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/snakeyaml-1.11.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/clj-time-0.4.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-lang-2.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-exec-1.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/kryo-2.21.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/clj-stacktrace-0.2.2.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-fileupload-1.2.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/logback-core-1.0.13.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/servlet-api-2.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/hiccup-0.3.6.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/json-simple-1.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/reflectasm-1.07-shaded.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/logback-classic-1.0.13.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-logging-1.1.3.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/tools.logging-0.2.3.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/ring-jetty-adapter-0.3.11.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/minlog-1.2.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/ring-devel-0.3.11.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/jetty-6.1.26.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/carbonite-1.4.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/conf
-Dlogfile.name=nimbus.log
-Dlogback.configurationFile=/opt/cloudera/parcels/STORM-0.9.5-u1/logback/cluster.xml
backtype.storm.daemon.nimbus
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:195)
at backtype.storm.util$loading__4910__auto__.invoke(util.clj:17)
at backtype.storm.util__init.load(Unknown Source)
at backtype.storm.util__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at clojure.lang.RT.loadClassForName(RT.java:2098)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5336)
at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
at clojure.core$load_lib.doInvoke(core.clj:5374)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:619)
at clojure.core$load_libs.doInvoke(core.clj:5417)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:621)
at clojure.core$use.doInvoke(core.clj:5507)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at
backtype.storm.scheduler.DefaultScheduler$loading__4910__auto__.invoke(DefaultScheduler.clj:16)
at backtype.storm.scheduler.DefaultScheduler__init.load(Unknown
Source)
at
backtype.storm.scheduler.DefaultScheduler__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at clojure.lang.RT.loadClassForName(RT.java:2098)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5336)
at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
at clojure.core$load_lib.doInvoke(core.clj:5374)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:619)
at clojure.core$load_libs.doInvoke(core.clj:5413)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:621)
at clojure.core$use.doInvoke(core.clj:5507)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at
backtype.storm.daemon.nimbus$loading__4910__auto__.invoke(nimbus.clj:16)
at backtype.storm.daemon.nimbus__init.load(Unknown Source)
at backtype.storm.daemon.nimbus__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at clojure.lang.RT.loadClassForName(RT.java:2098)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:415)
at backtype.storm.daemon.nimbus.<clinit>(Unknown Source)
Caused by: java.lang.RuntimeException: Could not find config file on classpath
/etc/storm/conf/storm_nimbus.yaml
at
backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:190)
at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
... 57 more
From: Garcia-Contractor, Joseph (CORP) [mailto:[email protected]]
Sent: Thursday, October 08, 2015 3:34 PM
To: [email protected]
Subject: RE: Storm and enviroment variables
Yeap… it certainly does… I missed that. Thanks for pointing it out! ☺
From: Priyank Shah [mailto:[email protected]]
Sent: Thursday, October 08, 2015 12:51 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Storm and enviroment variables
Glad that it fixed your issue. The python script has the —config option. Check
https://github.com/apache/storm/blob/master/bin/storm.py#L632 out.
From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "[email protected]<mailto:[email protected]>"
Date: Thursday, October 8, 2015 at 6:52 AM
To: "[email protected]<mailto:[email protected]>"
Subject: RE: Storm and enviroment variables
I did not see a –config option in the storm python start script… I did however
find in there that if I set the STORM_CONF_DIR environment variable it would
look for storm.yaml in that dir and this fixed my issue.
From: Priyank Shah [mailto:[email protected]]
Sent: Wednesday, October 07, 2015 5:31 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Storm and enviroment variables
I think you can specify a config file that is other than the default. Can you
try —config <pathToYourYamlFile> command line option while running storm
command ?
From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "[email protected]<mailto:[email protected]>"
Date: Wednesday, October 7, 2015 at 2:15 PM
To: "[email protected]<mailto:[email protected]>"
Subject: RE: Storm and enviroment variables
Ok so an additional question here…
Can I tell storm on startup to use a storm.yaml file that I specify with a full
path, or will it only look in /etc/storm/conf/storm.yaml?
Joe
From: Priyank Shah [mailto:[email protected]]
Sent: Wednesday, September 30, 2015 2:31 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Storm and enviroment variables
Hi Joseph,
I think storm core components don’t support environment variables. It either
has to be a config yaml file or a –c option. For –c option, its a key value
pair of the form key=value. I checked the code and key should be same as key in
yaml. It should work for simple strings. But not sure about other types. In
the code its trying to parse it as a json. You can give it a try.
I don’t know about logback file . But you can configure your log directory
using –c storm.log.dir=/var/log/ option.
From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "[email protected]<mailto:[email protected]>"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "[email protected]<mailto:[email protected]>"
Subject: Storm and enviroment variables
Hi All,
Another question.
Does storm accept environment variables that can be used in place of
/etc/storm/conf/storm.yaml?
If so, what is the format of the environment variable it? IE.: storm.local.dir
in storm.yaml would be STORM_LOCAL_DIR in the environment.
I see that I can use –c at the command line but I would prefer using
environment variable without having to build the command line args. If I need
to go with –c, what would the arguments still be in yaml format? IE.:
storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]
Is there a way to override the where the logback file is located and read from?
________________________________
This message and any attachments are intended only for the use of the addressee
and may contain information that is privileged and confidential. If the reader
of the message is not the intended recipient or an authorized representative of
the intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication
in error, notify the sender immediately by return email and delete the message
and any attachments from your system.
