Okay, in your development environment you are using 1.0.1, but your deployment environment is 0.9.6. There's a lot of distance between those versions, and I've been looking at the wrong code.
Before I go back to the 0.9.6 code, are you saying you're seeing this behavior with both versions? -Taylor > On Oct 7, 2016, at 4:00 PM, Brad Rhodes <[email protected]> wrote: > > I am asking again for a bit of help. I would not re-ask, but I have spent a > ton of time on this, debugging the startup code in nimbus and I am somewhat > stuck. > > I can provide logs and traces showing that supervisor is getting Zookeeper > java.class.path (server or client) and using that to start the worker. Let > me know if you want to see them. > > I know these values are getting sent to zookeeper in the > org.apache.curator.framework : CuratorFrameworkFactory class on line 61: > Public static Builder builder() {return new Builder();} > > I know the config processing is largely happening in > storm-core/src/clj/backtype.storm/util.clj and config.clj. With some > happening in storm-core/src/jvm/utils/Utils.java > > I know that the initial set up of the config structure is done in Utils.java > line 70. > > Static{ > Map conf = readStormConfig() > serializationDelegate = getSerializationDelegate(conf); > } > > I just cannot find where in these files someone is taking the javaagent line > and appending it to the classpath. It seems somewhere between the initial > config read, and when this is sent to zookeeper by the Builder(); call. > > I could use any help, like if anyone is some pointers on how to debug Clojure > startup code with java static init <cinit>. I am using the Cursive Debugger > add-in for Intellij. But I think the issue here is more the static{} > functions that are running in java. This all makes it super slow debugging. > Sometimes you end up stepping through bytecode, and not getting variables. > So if anyone has pointers on how they work in these areas, that would be a > huge help. > > Thanks again for any help. > brad > > > > > > From: Brad Rhodes [mailto:[email protected]] > Sent: Wednesday, October 5, 2016 2:15 PM > To: [email protected] > Subject: RE: Nimbus setting java.class.path to include javaagent > > Hey Taylor, > Sorry, that path was from my development machine. > > On the nimbus server in question storm.classpath returns: > /opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar > > From: [email protected] > To: [email protected] > Subject: RE: Nimbus setting java.class.path to include javaagent > Date: Wed, 5 Oct 2016 12:52:20 -0700 > > That may be the case. > > I have confirmed that supervisor is getting either the Client classpath or > Server classpath sent to zookeeper to start worker processes. If I edit the > config file on the supervisor, the workers still start with the classpath > that is on the nimbus server, with the javaagent included in the classpath. > The only way that seems possible is that supervisor is starting with a > classpath it has gotten from zookeeper. > > Yes, the weird part is that the agent is sneaking onto the class path when > the worker starts up. > > storm classpath returns: > /usr/local/Cellar/storm/1.0.1/libexec/lib/asm-5.0.3.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/clojure-1.7.0.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/disruptor-3.3.2.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/kryo-3.0.3.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-api-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-core-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-slf4j-impl-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/minlog-1.3.0.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/objenesis-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/reflectasm-1.10.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/servlet-api-2.5.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/slf4j-api-1.7.7.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/storm-core-1.0.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/storm-rename-hack-1.0.1.jar > > thanks for all of your help. > brad > > Subject: Re: Nimbus setting java.class.path to include javaagent > From: [email protected] > Date: Wed, 5 Oct 2016 15:33:47 -0400 > To: [email protected] > > What your seeing is the ZooKeeper client running within Nimbus/Supervisor > just printing out it’s class path, which it inherits from Storm’s class path. > It’s not really sending that to ZooKeeper. > > What’s weird is that the agent jar sneaking onto the class path. > > On your supervisor node, can you run `storm classpath` and see what it prints? > > -Taylor > > > On Oct 5, 2016, at 1:17 PM, Brad Rhodes <[email protected]> wrote: > > Storm 0.9.6. We are working on moving to 1.0.2 as we speak. > > nimbus.childopts: "-Xmx1024m -DjmxRegistryPort=1098 -DjmxServerPort=1099 > -javaagent:/opt/monitoring_agent/bobbrad.jar > -agentlib:jdwp=transport=dt_socket,server=y,address=8100,suspend= > y -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties > -Dapplication.name=storm.test.lodgingamenity.nimbus" > > > supervisor.childopts: "-Xmx256m -DjmxRegistryPort=1098 -DjmxServerPort=1099 > -javaagent:/opt/monitoring_agent/bobbrad2.jar > -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring > .properties -Dapplication.name=storm.test.lodgingamenity.supervisor" > > > worker.childopts: "-Xmx1024m > -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties > -Dapplication.name=storm.test.lodgingamenity.worker.%ID%" > > > From: P. Taylor Goetz [mailto:[email protected]] > Sent: Tuesday, October 4, 2016 9:46 AM > To: [email protected] > Subject: Re: Nimbus setting java.class.path to include javaagent > > Can you post your values for nimbus.childopts, supervisor.childopts, and > worker.childopts? Also what version of Storm are you on? > > -Taylor > > On Sep 28, 2016, at 6:06 PM, Brad Rhodes <[email protected]> wrote: > > I have a situation where we have a javaagent jar file. Someone has added a > number of classes to the corporate standard javaagent jar file. Nimbus > appears to append -javaagent jar files to the -cp (claspath) for the > supervisor process. > > When Nimbus starts up java is started with a command like: > > /usr/java/latest/bin/java -server -Dstorm.options= -Dstorm.home=/opt/storm > -Dstorm.log.dir=/opt/storm/logs > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= > -cp > /opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf > -Xmx1024m -DjmxRegistryPort=1098 -DjmxServerPort=1099 > -javaagent:/opt/monitoring_agent/bobbrad.jar > -agentlib:jdwp=transport=dt_socket,server=y,address=8100,suspend=y > -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties > -Dapplication.name=storm.test.lodgingamenity.nimbus -Dlogfile.name=nimbus.log > -Dlogback.configurationFile=/opt/storm/logback/cluster.xml > backtype.storm.daemon.nimbus > > Nimbus at some point sends the following to Zookeeper: > 016-09-28T18:50:45.627+0000 o.a.s.z.ZooKeeper [INFO] Client > environment:java.class.path=/opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf:/opt/monitoring_agent/bobbrad.jar > > and > Server > environment:java.class.path=/opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf:/opt/monitoring_agent/bobbrad.jar > > When the supervisor starts up you will see bobbrad.jar in the classpath. > > I see the code in supervisor.clj > (defn resources-jar [] > (->> (.split (current-classpath) File/pathSeparator) > (filter #(.endsWith % ".jar")) > (filter #(zip-contains-dir? % RESOURCES-SUBDIR)) > first )) > It does not seem this code is the source of this running with zookeeper > cluster. Suprivisor is applying what comes down from Zookeeper > java.classpath. > Can someone point me to where that is happening in a cluster environment.? > Thanks for any help. > >
