I do not see any difference from 1.01 to 0.9.6 in this regard. But I do not have a cluster of 1.01 setup yet, so that is development environment and running local, which is different.
From: P. Taylor Goetz [mailto:[email protected]] Sent: Friday, October 7, 2016 3:26 PM To: [email protected] Subject: Re: Nimbus setting java.class.path to include javaagent Okay, in your development environment you are using 1.0.1, but your deployment environment is 0.9.6. There's a lot of distance between those versions, and I've been looking at the wrong code. Before I go back to the 0.9.6 code, are you saying you're seeing this behavior with both versions? -Taylor On Oct 7, 2016, at 4:00 PM, Brad Rhodes <[email protected] <mailto:[email protected]> > wrote: I am asking again for a bit of help. I would not re-ask, but I have spent a ton of time on this, debugging the startup code in nimbus and I am somewhat stuck. I can provide logs and traces showing that supervisor is getting Zookeeper java.class.path (server or client) and using that to start the worker. Let me know if you want to see them. I know these values are getting sent to zookeeper in the org.apache.curator.framework : CuratorFrameworkFactory class on line 61: Public static Builder builder() {return new Builder();} I know the config processing is largely happening in storm-core/src/clj/backtype.storm/util.clj and config.clj. With some happening in storm-core/src/jvm/utils/Utils.java I know that the initial set up of the config structure is done in Utils.java line 70. Static{ Map conf = readStormConfig() serializationDelegate = getSerializationDelegate(conf); } I just cannot find where in these files someone is taking the javaagent line and appending it to the classpath. It seems somewhere between the initial config read, and when this is sent to zookeeper by the Builder(); call. I could use any help, like if anyone is some pointers on how to debug Clojure startup code with java static init <cinit>. I am using the Cursive Debugger add-in for Intellij. But I think the issue here is more the static{} functions that are running in java. This all makes it super slow debugging. Sometimes you end up stepping through bytecode, and not getting variables. So if anyone has pointers on how they work in these areas, that would be a huge help. Thanks again for any help. brad From: Brad Rhodes [mailto:[email protected]] Sent: Wednesday, October 5, 2016 2:15 PM To: [email protected] <mailto:[email protected]> Subject: RE: Nimbus setting java.class.path to include javaagent Hey Taylor, Sorry, that path was from my development machine. On the nimbus server in question storm.classpath returns: /opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar _____ From: [email protected] <mailto:[email protected]> To: [email protected] <mailto:[email protected]> Subject: RE: Nimbus setting java.class.path to include javaagent Date: Wed, 5 Oct 2016 12:52:20 -0700 That may be the case. I have confirmed that supervisor is getting either the Client classpath or Server classpath sent to zookeeper to start worker processes. If I edit the config file on the supervisor, the workers still start with the classpath that is on the nimbus server, with the javaagent included in the classpath. The only way that seems possible is that supervisor is starting with a classpath it has gotten from zookeeper. Yes, the weird part is that the agent is sneaking onto the class path when the worker starts up. storm classpath returns: /usr/local/Cellar/storm/1.0.1/libexec/lib/asm-5.0.3.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/clojure-1.7.0.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/disruptor-3.3.2.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/kryo-3.0.3.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-api-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-core-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-slf4j-impl-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/minlog-1.3.0.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/objenesis-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/reflectasm-1.10.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/servlet-api-2.5.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/slf4j-api-1.7.7.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/storm-core-1.0.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/storm-rename-hack-1.0.1.jar thanks for all of your help. brad _____ Subject: Re: Nimbus setting java.class.path to include javaagent From: [email protected] <mailto:[email protected]> Date: Wed, 5 Oct 2016 15:33:47 -0400 To: [email protected] <mailto:[email protected]> What your seeing is the ZooKeeper client running within Nimbus/Supervisor just printing out it’s class path, which it inherits from Storm’s class path. It’s not really sending that to ZooKeeper. What’s weird is that the agent jar sneaking onto the class path. On your supervisor node, can you run `storm classpath` and see what it prints? -Taylor On Oct 5, 2016, at 1:17 PM, Brad Rhodes <[email protected] <mailto:[email protected]> > wrote: Storm 0.9.6. We are working on moving to 1.0.2 as we speak. nimbus.childopts: "-Xmx1024m -DjmxRegistryPort=1098 -DjmxServerPort=1099 -javaagent:/opt/monitoring_agent/bobbrad.jar -agentlib:jdwp=transport=dt_socket,server=y,address=8100,suspend= y -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties -Dapplication.name=storm.test.lodgingamenity.nimbus" supervisor.childopts: "-Xmx256m -DjmxRegistryPort=1098 -DjmxServerPort=1099 -javaagent:/opt/monitoring_agent/bobbrad2.jar -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring .properties -Dapplication.name=storm.test.lodgingamenity.supervisor" worker.childopts: "-Xmx1024m -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties -Dapplication.name=storm.test.lodgingamenity.worker.%ID%" From: P. Taylor Goetz [mailto:[email protected]] Sent: Tuesday, October 4, 2016 9:46 AM To: [email protected] <mailto:[email protected]> Subject: Re: Nimbus setting java.class.path to include javaagent Can you post your values for nimbus.childopts, supervisor.childopts, and worker.childopts? Also what version of Storm are you on? -Taylor On Sep 28, 2016, at 6:06 PM, Brad Rhodes <[email protected] <mailto:[email protected]> > wrote: I have a situation where we have a javaagent jar file. Someone has added a number of classes to the corporate standard javaagent jar file. Nimbus appears to append -javaagent jar files to the -cp (claspath) for the supervisor process. When Nimbus starts up java is started with a command like: /usr/java/latest/bin/java -server -Dstorm.options= -Dstorm.home=/opt/storm -Dstorm.log.dir=/opt/storm/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf -Xmx1024m -DjmxRegistryPort=1098 -DjmxServerPort=1099 -javaagent:/opt/monitoring_agent/bobbrad.jar -agentlib:jdwp=transport=dt_socket,server=y,address=8100,suspend=y -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties -Dapplication.name=storm.test.lodgingamenity.nimbus -Dlogfile.name=nimbus.log -Dlogback.configurationFile=/opt/storm/logback/cluster.xml backtype.storm.daemon.nimbus Nimbus at some point sends the following to Zookeeper: 016-09-28T18:50:45.627+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf:/opt/monitoring_agent/bobbrad.jar and Server environment:java.class.path=/opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf:/opt/monitoring_agent/bobbrad.jar When the supervisor starts up you will see bobbrad.jar in the classpath. I see the code in supervisor.clj (defn resources-jar [] (->> (.split (current-classpath) File/pathSeparator) (filter #(.endsWith % ".jar")) (filter #(zip-contains-dir? % RESOURCES-SUBDIR)) first )) It does not seem this code is the source of this running with zookeeper cluster. Suprivisor is applying what comes down from Zookeeper java.classpath. Can someone point me to where that is happening in a cluster environment.? Thanks for any help.
