Sorry for the delay here guys - I've added the -w 60 param to /usr/local/monit/stop_*_topology.sh and I'm still seeing threads get left behind during reboots.
Jon On Wed, Oct 5, 2016 at 3:54 PM [email protected] <[email protected]> wrote: > Right, I will try that. I guess my main question, though, is why would > that need more time? Is it a function of CPU, memory, disk, or otherwise? > Obviously if the timeout is the issue it's probably related to what I'm > sending into the cluster and how resources are allocated in relation to the > workload but I plan to scale this up roughly 10x and don't want to end up > with a 600 second timeout. > > Jon > > On Wed, Oct 5, 2016, 15:34 Otto Fowler <[email protected]> wrote: > > What you would have to do to try this is to edit > /usr/local/monit/stop_bro_topology.sh and add in a -w parameter. > to read: > > storm kill bro -w 60 > > save that and then try to stop if from monit again. > > If that works, then we will know. > > On October 5, 2016 at 15:18:43, [email protected] ([email protected]) wrote: > > topology.message.timeout.secs is set to 30 according to ambari. > > What you requested is essentially what I did before, except instead of > using `storm kill` I used monit to kill all of the topologies, and I > waited > multiple minutes before stopping nimbus/supervisor/drpc, then waited a > couple more minutes before grabbing the prior metrics/processes. I've seen > this multiple times across the past week and each time I see it I shut > everything down, do a `kill -9` on the storm processes left behind, make > sure no processes are running as the storm user on my entire cluster, then > bring things up in ambari, then monit once ambari says > nimbus/supervisor/drpc is up. > > Jon > > On Wed, Oct 5, 2016 at 3:09 PM P. Taylor Goetz <[email protected]> wrote: > > > To build on what Otto said: > > > > What’s your topology.message.timeout.secs configuration value? The > default > > is 30. > > > > If you issue the `storm kill` command without the -w option, Storm will > > first deactivate the topology, then wait topology.message.timeout.secs > > before actually shutting down the topology. > > > > Also, if you have topologies deployed, and stop ninmubs, supervisors, > etc. > > the workers may continue to run (depending on how you have monit setup). > > > > Can you try the following: > > > > 1. Kill the topology and wait topology.message.timeout.sec so Storm > > properly shuts down the topology. > > 2. Stop nimbus/supervisor/etc. > > 3. Check to see if you still have threads left behind. > > > > > > -Taylor > > > > > On Oct 5, 2016, at 2:31 PM, Otto Fowler <[email protected]> > wrote: > > > > > > The shutdown topology scripts only do a storm kill <item>. > > > They probably need to -w WAITTIME. > > > > > > > > > -- > > > > > > Sent with Airmail > > > > > > On October 5, 2016 at 13:53:33, [email protected] ([email protected]) > > wrote: > > > > > > Hi All, > > > > > > While tuning Metron I have been doing a large amount of storm topology > > > restarts, as I tune the parallelism and memory use of bolts. I have > > > noticed multiple times that some threads will be left behind on > restart, > > > which eventually causes all topologies to fail to start due to > > > "java.lang.OutOfMemoryError: unable to create new native thread". I am > > > only using the bro, indexing, and enrichments topologies right now, > and I > > > have seen all of them leave hanging processes behind. Has anybody else > > > seen anything like this? > > > > > > I'm exclusively using monit to do topology restarts, and Ambari to > > restart > > > supervisors/nimbus/UI/DRPC processes. When it is currently registering > as > > > stopped in both monit and Ambari, I see the following threads left > > behind: > > >> [jzeolla@node1 storm]$ ps -austorm | wc -l > > >> 6 > > >> [jzeolla@node2 storm]$ ps -austorm | wc -l > > >> 12 > > >> [jzeolla@node3 storm]$ ps -austorm | wc -l > > >> 9 > > >> [jzeolla@node4 storm]$ ps -austorm | wc -l > > >> 11 > > >> [jzeolla@node5 storm]$ ps -austorm | wc -l > > >> 9 > > > > > > Some sample processes: > > >> storm 14257 0.1 0.2 6279184 293912 ? Sl Oct03 3:44 > > > /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer > > > -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > > > -Dstorm.log.dir=/var/log/storm > > > > > > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > > > > > -Dstorm.conf.file= -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > > > > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf > > > > > > > -Xmx128m -Dlogfile.name=logviewer.log > > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml > > > backtype.storm.daemon.logviewer > > >> storm 32072 0.0 0.1 37337056 177112 ? Sl Oct02 2:02 > > > /usr/jdk64/jdk1.8.0_60/bin/java -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > > > > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.id=bro-13-1475456606 > > > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 > > > -Dstorm.log.dir=/var/log/storm > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > -Xmx20G > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > > > > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > > > > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > > -Dstorm.options= > > > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > -Dstorm.id=bro-13-1475456606 > > > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 > -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > > > > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > backtype.storm.daemon.worker bro-13-1475456606 > > > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6704 > > > f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 > > >> storm 32127 121 7.4 32422464 9799588 ? Sl Oct02 4337:42 > > > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > > > > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > > > > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > > -Dstorm.options= > > > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > -Dstorm.id=bro-13-1475456606 > > > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 > -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > > > > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > backtype.storm.daemon.worker bro-13-1475456606 > > > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6704 > > > f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 > > >> storm 47882 0.0 0.1 37338556 188920 ? Sl Oct04 0:47 > > > /usr/jdk64/jdk1.8.0_60/bin/java -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > > > > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > -Dlogfile.name=indexing-16-1475589543-worker-6700.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > -Dstorm.id=indexing-16-1475589543 > > > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 > > > -Dstorm.log.dir=/var/log/storm > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > -Xmx2G > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > > > > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > > > > > -Dlogfile.name=indexing-16-1475589543-worker-6700.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > > -Dstorm.options= > > > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > -Dstorm.id=indexing-16-1475589543 > > > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 > -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > > > > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > backtype.storm.daemon.worker indexing-16-1475589543 > > > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6700 > > > 5e17c5c3-e6ab-4b48-a48a-dc540cda7087 > > >> storm 47978 4.7 0.5 19996036 740372 ? Sl Oct04 65:04 > > > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx2G > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > > > > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > > > > > -Dlogfile.name=indexing-16-1475589543-worker-6700.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > > -Dstorm.options= > > > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > -Dstorm.id=indexing-16-1475589543 > > > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 > -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > > > > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > backtype.storm.daemon.worker indexing-16-1475589543 > > > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6700 > > > 5e17c5c3-e6ab-4b48-a48a-dc540cda7087 > > >> storm 21301 0.1 0.2 6347824 298632 ? Sl Oct03 3:44 > > > /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer > > > -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > > > -Dstorm.log.dir=/var/log/storm > > > > > > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > > > > > -Dstorm.conf.file= -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > > > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf > > > > > > > -Xmx128m -Dlogfile.name=logviewer.log > > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml > > > backtype.storm.daemon.logviewer > > >> storm 14046 102 19.3 42125944 25539856 ? Sl Oct03 2655:13 > > > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > > > > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > > > > > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > > -Dstorm.options= > > > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > -Dstorm.id=enrichment-12-1475456272 > > > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 > -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > > > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > backtype.storm.daemon.worker enrichment-12-1475456272 > > > eff7f971-ebe2-4f53-beab-99b061bfde10 6700 > > > 5d586063-e405-4a67-a644-53a0ac943ab0 > > >> storm 6544 0.0 0.1 37339132 180872 ? Sl Oct02 3:22 > > > /usr/jdk64/jdk1.8.0_60/bin/java -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > > > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.id=bro-13-1475456606 > > > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 > > > -Dstorm.log.dir=/var/log/storm > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > -Xmx20G > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > > > > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > > > > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > > -Dstorm.options= > > > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > -Dstorm.id=bro-13-1475456606 > > > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 > -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > > > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > backtype.storm.daemon.worker bro-13-1475456606 > > > eff7f971-ebe2-4f53-beab-99b061bfde10 6704 > > > f917291c-bd1e-4a9c-a52d-b9c5fef0567e > > >> storm 6592 192 9.8 34986568 12987704 ? Sl Oct02 6866:30 > > > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > > > > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > > > > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > > -Dstorm.options= > > > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > -Dstorm.id=bro-13-1475456606 > > > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 > -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > > > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > backtype.storm.daemon.worker bro-13-1475456606 > > > eff7f971-ebe2-4f53-beab-99b061bfde10 6704 > > > f917291c-bd1e-4a9c-a52d-b9c5fef0567e > > >> storm 13998 0.0 0.1 37339104 184140 ? Sl Oct03 1:57 > > > /usr/jdk64/jdk1.8.0_60/bin/java -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > > > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > > -Dstorm.id=enrichment-12-1475456272 > > > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 > > > -Dstorm.log.dir=/var/log/storm > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > -Xmx20G > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > > > > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > > > > > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log > > > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > > -Dstorm.options= > > > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > > > > > -Dstorm.id=enrichment-12-1475456272 > > > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 > -cp > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > > > > > 2.4.0. > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > > > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > > > > > backtype.storm.daemon.worker enrichment-12-1475456272 > > > eff7f971-ebe2-4f53-beab-99b061bfde10 6700 > > > 5d586063-e405-4a67-a644-53a0ac943ab0 > > > > > > > > > Logs showing the inability to start a topology: > > >> 2016-10-04 10:10:57.787 b.s.m.n.Client [INFO] creating Netty Client, > > > connecting to pgh-dat-t53.iso.local.cmu.edu:6702, bufferSize: 5242880 > > >> 2016-10-04 10:10:57.787 o.a.s.c.r.ExponentialBackoffRetry [WARN] > > > maxRetries too large (30). Pinning to 29 > > >> 2016-10-04 10:10:57.787 b.s.u.StormBoundedExponentialBackoffRetry > [INFO] > > > The baseSleepTimeMs [100] the maxSleepTimeMs [1000] the maxRetries > [30] > > >> 2016-10-04 10:10:57.787 b.s.m.n.Client [INFO] connection attempt 1 to > > > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6702 > scheduled > > to > > > run in 0 ms > > >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established > to > > > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6703 > > >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established > to > > > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6702 > > >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established > to > > > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6701 > > >> 2016-10-04 10:10:58.030 b.s.d.executor [INFO] Loading executor > > > enrichmentJoinBolt:[2 2] > > >> 2016-10-04 10:10:58.064 b.s.d.executor [INFO] Loaded executor tasks > > > enrichmentJoinBolt:[2 2] > > >> 2016-10-04 10:10:58.071 b.s.d.executor [INFO] Finished loading > executor > > > enrichmentJoinBolt:[2 2] > > >> 2016-10-04 10:10:58.081 b.s.d.executor [INFO] Loading executor > > > simpleHBaseThreatIntelBolt:[22 22] > > >> 2016-10-04 10:10:58.095 b.s.d.executor [INFO] Loaded executor tasks > > > simpleHBaseThreatIntelBolt:[22 22] > > >> 2016-10-04 10:10:58.097 b.s.d.executor [INFO] Finished loading > executor > > > simpleHBaseThreatIntelBolt:[22 22] > > >> 2016-10-04 10:10:58.104 b.s.d.executor [INFO] Loading executor > > > kafkaSpout:[18 18] > > >> 2016-10-04 10:10:58.112 b.s.d.executor [INFO] Loaded executor tasks > > > kafkaSpout:[18 18] > > >> 2016-10-04 10:10:58.118 b.s.d.executor [INFO] Finished loading > executor > > > kafkaSpout:[18 18] > > >> 2016-10-04 10:10:58.125 b.s.d.executor [INFO] Loading executor > > > geoEnrichmentBolt:[6 6] > > >> 2016-10-04 10:10:58.135 b.s.d.executor [INFO] Loaded executor tasks > > > geoEnrichmentBolt:[6 6] > > >> 2016-10-04 10:10:58.137 b.s.d.executor [ERROR] > > >> java.lang.OutOfMemoryError: unable to create new native thread > > >> at java.lang.Thread.start0(Native Method) ~[?:1.8.0_60] > > >> at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_60] > > >> at backtype.storm.util$async_loop.doInvoke(util.clj:491) > > > ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.RestFn.invoke(RestFn.java:559) > > > ~[clojure-1.6.0.jar:?] > > >> at backtype.storm.daemon.executor$fn__7245.invoke(executor.clj:820) > > > ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.MultiFn.invoke(MultiFn.java:236) > > > ~[clojure-1.6.0.jar:?] > > >> at > > > > > > backtype.storm.daemon.executor$mk_executor$fn__7116.invoke(executor.clj:351) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at > > > backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:350) > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721$iter__7726__7730$fn__7731.invoke(worker.clj:541) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.6.0.jar:?] > > >> at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.6.0.jar:?] > > >> at clojure.lang.Cons.next(Cons.java:39) [clojure-1.6.0.jar:?] > > >> at clojure.lang.RT.next(RT.java:598) [clojure-1.6.0.jar:?] > > >> at clojure.core$next.invoke(core.clj:64) [clojure-1.6.0.jar:?] > > >> at clojure.core$dorun.invoke(core.clj:2856) [clojure-1.6.0.jar:?] > > >> at clojure.core$doall.invoke(core.clj:2871) [clojure-1.6.0.jar:?] > > >> at > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721.run(worker.clj:541) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at java.security.AccessController.doPrivileged(Native Method) > > > ~[?:1.8.0_60] > > >> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_60] > > >> at > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto____7720.invoke(worker.clj:515) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.AFn.applyToHelper(AFn.java:178) > > > [clojure-1.6.0.jar:?] > > >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] > > >> at clojure.core$apply.invoke(core.clj:624) [clojure-1.6.0.jar:?] > > >> at > > > > > > backtype.storm.daemon.worker$fn__7719$mk_worker__7805.doInvoke(worker.clj:491) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:?] > > >> at backtype.storm.daemon.worker$_main.invoke(worker.clj:641) > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.AFn.applyToHelper(AFn.java:165) > > > [clojure-1.6.0.jar:?] > > >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] > > >> at backtype.storm.daemon.worker.main(Unknown Source) > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> 2016-10-04 10:10:58.169 b.s.util [ERROR] Halting process: ("Worker > > died") > > >> java.lang.RuntimeException: ("Worker died") > > >> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:332) > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?] > > >> at > > > backtype.storm.daemon.worker$fn__7818$fn__7819.invoke(worker.clj:636) > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at > > > > > > backtype.storm.daemon.executor$mk_executor_data$fn__7078$fn__7079.invoke(executor.clj:256) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at > > > > > > backtype.storm.daemon.executor$mk_executor$fn__7116.invoke(executor.clj:350) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at > > > backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:350) > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721$iter__7726__7730$fn__7731.invoke(worker.clj:541) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.6.0.jar:?] > > >> at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.6.0.jar:?] > > >> at clojure.lang.Cons.next(Cons.java:39) [clojure-1.6.0.jar:?] > > >> at clojure.lang.RT.next(RT.java:598) [clojure-1.6.0.jar:?] > > >> at clojure.core$next.invoke(core.clj:64) [clojure-1.6.0.jar:?] > > >> at clojure.core$dorun.invoke(core.clj:2856) [clojure-1.6.0.jar:?] > > >> at clojure.core$doall.invoke(core.clj:2871) [clojure-1.6.0.jar:?] > > >> at > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721.run(worker.clj:541) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at java.security.AccessController.doPrivileged(Native Method) > > > ~[?:1.8.0_60] > > >> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_60] > > >> at > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto____7720.invoke(worker.clj:515) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.AFn.applyToHelper(AFn.java:178) > > > [clojure-1.6.0.jar:?] > > >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] > > >> at clojure.core$apply.invoke(core.clj:624) [clojure-1.6.0.jar:?] > > >> at > > > > > > backtype.storm.daemon.worker$fn__7719$mk_worker__7805.doInvoke(worker.clj:491) > > > > > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:?] > > >> at backtype.storm.daemon.worker$_main.invoke(worker.clj:641) > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > >> at clojure.lang.AFn.applyToHelper(AFn.java:165) > > > [clojure-1.6.0.jar:?] > > >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] > > >> at backtype.storm.daemon.worker.main(Unknown Source) > > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > > > -- > > > > > > Jon > > > > -- > > Jon > > -- > > Jon > -- Jon
