To build on what Otto said: What’s your topology.message.timeout.secs configuration value? The default is 30.
If you issue the `storm kill` command without the -w option, Storm will first deactivate the topology, then wait topology.message.timeout.secs before actually shutting down the topology. Also, if you have topologies deployed, and stop ninmubs, supervisors, etc. the workers may continue to run (depending on how you have monit setup). Can you try the following: 1. Kill the topology and wait topology.message.timeout.sec so Storm properly shuts down the topology. 2. Stop nimbus/supervisor/etc. 3. Check to see if you still have threads left behind. -Taylor > On Oct 5, 2016, at 2:31 PM, Otto Fowler <[email protected]> wrote: > > The shutdown topology scripts only do a storm kill <item>. > They probably need to -w WAITTIME. > > > -- > > Sent with Airmail > > On October 5, 2016 at 13:53:33, [email protected] ([email protected]) wrote: > > Hi All, > > While tuning Metron I have been doing a large amount of storm topology > restarts, as I tune the parallelism and memory use of bolts. I have > noticed multiple times that some threads will be left behind on restart, > which eventually causes all topologies to fail to start due to > "java.lang.OutOfMemoryError: unable to create new native thread". I am > only using the bro, indexing, and enrichments topologies right now, and I > have seen all of them leave hanging processes behind. Has anybody else > seen anything like this? > > I'm exclusively using monit to do topology restarts, and Ambari to restart > supervisors/nimbus/UI/DRPC processes. When it is currently registering as > stopped in both monit and Ambari, I see the following threads left behind: >> [jzeolla@node1 storm]$ ps -austorm | wc -l >> 6 >> [jzeolla@node2 storm]$ ps -austorm | wc -l >> 12 >> [jzeolla@node3 storm]$ ps -austorm | wc -l >> 9 >> [jzeolla@node4 storm]$ ps -austorm | wc -l >> 11 >> [jzeolla@node5 storm]$ ps -austorm | wc -l >> 9 > > Some sample processes: >> storm 14257 0.1 0.2 6279184 293912 ? Sl Oct03 3:44 > /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer > -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > -Dstorm.log.dir=/var/log/storm > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > -Dstorm.conf.file= -cp > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf > > -Xmx128m -Dlogfile.name=logviewer.log > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml > backtype.storm.daemon.logviewer >> storm 32072 0.0 0.1 37337056 177112 ? Sl Oct02 2:02 > /usr/jdk64/jdk1.8.0_60/bin/java -cp > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.id=bro-13-1475456606 > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 > -Dstorm.log.dir=/var/log/storm > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= -Dstorm.options= > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > -Dstorm.id=bro-13-1475456606 > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 -cp > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > backtype.storm.daemon.worker bro-13-1475456606 > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6704 > f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 >> storm 32127 121 7.4 32422464 9799588 ? Sl Oct02 4337:42 > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= -Dstorm.options= > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > -Dstorm.id=bro-13-1475456606 > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 -cp > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > backtype.storm.daemon.worker bro-13-1475456606 > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6704 > f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 >> storm 47882 0.0 0.1 37338556 188920 ? Sl Oct04 0:47 > /usr/jdk64/jdk1.8.0_60/bin/java -cp > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > -Dlogfile.name=indexing-16-1475589543-worker-6700.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.id=indexing-16-1475589543 > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 > -Dstorm.log.dir=/var/log/storm > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx2G > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > -Dlogfile.name=indexing-16-1475589543-worker-6700.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= -Dstorm.options= > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > -Dstorm.id=indexing-16-1475589543 > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 -cp > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > backtype.storm.daemon.worker indexing-16-1475589543 > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6700 > 5e17c5c3-e6ab-4b48-a48a-dc540cda7087 >> storm 47978 4.7 0.5 19996036 740372 ? Sl Oct04 65:04 > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx2G > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > -Dlogfile.name=indexing-16-1475589543-worker-6700.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= -Dstorm.options= > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > -Dstorm.id=indexing-16-1475589543 > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 -cp > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > backtype.storm.daemon.worker indexing-16-1475589543 > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6700 > 5e17c5c3-e6ab-4b48-a48a-dc540cda7087 >> storm 21301 0.1 0.2 6347824 298632 ? Sl Oct03 3:44 > /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer > -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > -Dstorm.log.dir=/var/log/storm > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > -Dstorm.conf.file= -cp > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf > > -Xmx128m -Dlogfile.name=logviewer.log > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml > backtype.storm.daemon.logviewer >> storm 14046 102 19.3 42125944 25539856 ? Sl Oct03 2655:13 > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= -Dstorm.options= > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > -Dstorm.id=enrichment-12-1475456272 > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 -cp > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > backtype.storm.daemon.worker enrichment-12-1475456272 > eff7f971-ebe2-4f53-beab-99b061bfde10 6700 > 5d586063-e405-4a67-a644-53a0ac943ab0 >> storm 6544 0.0 0.1 37339132 180872 ? Sl Oct02 3:22 > /usr/jdk64/jdk1.8.0_60/bin/java -cp > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.id=bro-13-1475456606 > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 > -Dstorm.log.dir=/var/log/storm > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= -Dstorm.options= > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > -Dstorm.id=bro-13-1475456606 > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 -cp > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > backtype.storm.daemon.worker bro-13-1475456606 > eff7f971-ebe2-4f53-beab-99b061bfde10 6704 > f917291c-bd1e-4a9c-a52d-b9c5fef0567e >> storm 6592 192 9.8 34986568 12987704 ? Sl Oct02 6866:30 > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > -Dlogfile.name=bro-13-1475456606-worker-6704.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= -Dstorm.options= > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > -Dstorm.id=bro-13-1475456606 > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 -cp > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > backtype.storm.daemon.worker bro-13-1475456606 > eff7f971-ebe2-4f53-beab-99b061bfde10 6704 > f917291c-bd1e-4a9c-a52d-b9c5fef0567e >> storm 13998 0.0 0.1 37339104 184140 ? Sl Oct03 1:57 > /usr/jdk64/jdk1.8.0_60/bin/java -cp > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.id=enrichment-12-1475456272 > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 > -Dstorm.log.dir=/var/log/storm > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= -Dstorm.options= > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > -Dstorm.id=enrichment-12-1475456272 > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 -cp > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > 2.4.0. > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > backtype.storm.daemon.worker enrichment-12-1475456272 > eff7f971-ebe2-4f53-beab-99b061bfde10 6700 > 5d586063-e405-4a67-a644-53a0ac943ab0 > > > Logs showing the inability to start a topology: >> 2016-10-04 10:10:57.787 b.s.m.n.Client [INFO] creating Netty Client, > connecting to pgh-dat-t53.iso.local.cmu.edu:6702, bufferSize: 5242880 >> 2016-10-04 10:10:57.787 o.a.s.c.r.ExponentialBackoffRetry [WARN] > maxRetries too large (30). Pinning to 29 >> 2016-10-04 10:10:57.787 b.s.u.StormBoundedExponentialBackoffRetry [INFO] > The baseSleepTimeMs [100] the maxSleepTimeMs [1000] the maxRetries [30] >> 2016-10-04 10:10:57.787 b.s.m.n.Client [INFO] connection attempt 1 to > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6702 scheduled to > run in 0 ms >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established to > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6703 >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established to > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6702 >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established to > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6701 >> 2016-10-04 10:10:58.030 b.s.d.executor [INFO] Loading executor > enrichmentJoinBolt:[2 2] >> 2016-10-04 10:10:58.064 b.s.d.executor [INFO] Loaded executor tasks > enrichmentJoinBolt:[2 2] >> 2016-10-04 10:10:58.071 b.s.d.executor [INFO] Finished loading executor > enrichmentJoinBolt:[2 2] >> 2016-10-04 10:10:58.081 b.s.d.executor [INFO] Loading executor > simpleHBaseThreatIntelBolt:[22 22] >> 2016-10-04 10:10:58.095 b.s.d.executor [INFO] Loaded executor tasks > simpleHBaseThreatIntelBolt:[22 22] >> 2016-10-04 10:10:58.097 b.s.d.executor [INFO] Finished loading executor > simpleHBaseThreatIntelBolt:[22 22] >> 2016-10-04 10:10:58.104 b.s.d.executor [INFO] Loading executor > kafkaSpout:[18 18] >> 2016-10-04 10:10:58.112 b.s.d.executor [INFO] Loaded executor tasks > kafkaSpout:[18 18] >> 2016-10-04 10:10:58.118 b.s.d.executor [INFO] Finished loading executor > kafkaSpout:[18 18] >> 2016-10-04 10:10:58.125 b.s.d.executor [INFO] Loading executor > geoEnrichmentBolt:[6 6] >> 2016-10-04 10:10:58.135 b.s.d.executor [INFO] Loaded executor tasks > geoEnrichmentBolt:[6 6] >> 2016-10-04 10:10:58.137 b.s.d.executor [ERROR] >> java.lang.OutOfMemoryError: unable to create new native thread >> at java.lang.Thread.start0(Native Method) ~[?:1.8.0_60] >> at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_60] >> at backtype.storm.util$async_loop.doInvoke(util.clj:491) > ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.RestFn.invoke(RestFn.java:559) > ~[clojure-1.6.0.jar:?] >> at backtype.storm.daemon.executor$fn__7245.invoke(executor.clj:820) > ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.MultiFn.invoke(MultiFn.java:236) > ~[clojure-1.6.0.jar:?] >> at > backtype.storm.daemon.executor$mk_executor$fn__7116.invoke(executor.clj:351) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at > backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:350) > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721$iter__7726__7730$fn__7731.invoke(worker.clj:541) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.6.0.jar:?] >> at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.6.0.jar:?] >> at clojure.lang.Cons.next(Cons.java:39) [clojure-1.6.0.jar:?] >> at clojure.lang.RT.next(RT.java:598) [clojure-1.6.0.jar:?] >> at clojure.core$next.invoke(core.clj:64) [clojure-1.6.0.jar:?] >> at clojure.core$dorun.invoke(core.clj:2856) [clojure-1.6.0.jar:?] >> at clojure.core$doall.invoke(core.clj:2871) [clojure-1.6.0.jar:?] >> at > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721.run(worker.clj:541) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at java.security.AccessController.doPrivileged(Native Method) > ~[?:1.8.0_60] >> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_60] >> at > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto____7720.invoke(worker.clj:515) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.AFn.applyToHelper(AFn.java:178) > [clojure-1.6.0.jar:?] >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] >> at clojure.core$apply.invoke(core.clj:624) [clojure-1.6.0.jar:?] >> at > backtype.storm.daemon.worker$fn__7719$mk_worker__7805.doInvoke(worker.clj:491) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:?] >> at backtype.storm.daemon.worker$_main.invoke(worker.clj:641) > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.AFn.applyToHelper(AFn.java:165) > [clojure-1.6.0.jar:?] >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] >> at backtype.storm.daemon.worker.main(Unknown Source) > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> 2016-10-04 10:10:58.169 b.s.util [ERROR] Halting process: ("Worker died") >> java.lang.RuntimeException: ("Worker died") >> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:332) > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?] >> at > backtype.storm.daemon.worker$fn__7818$fn__7819.invoke(worker.clj:636) > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at > backtype.storm.daemon.executor$mk_executor_data$fn__7078$fn__7079.invoke(executor.clj:256) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at > backtype.storm.daemon.executor$mk_executor$fn__7116.invoke(executor.clj:350) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at > backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:350) > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721$iter__7726__7730$fn__7731.invoke(worker.clj:541) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.6.0.jar:?] >> at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.6.0.jar:?] >> at clojure.lang.Cons.next(Cons.java:39) [clojure-1.6.0.jar:?] >> at clojure.lang.RT.next(RT.java:598) [clojure-1.6.0.jar:?] >> at clojure.core$next.invoke(core.clj:64) [clojure-1.6.0.jar:?] >> at clojure.core$dorun.invoke(core.clj:2856) [clojure-1.6.0.jar:?] >> at clojure.core$doall.invoke(core.clj:2871) [clojure-1.6.0.jar:?] >> at > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721.run(worker.clj:541) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at java.security.AccessController.doPrivileged(Native Method) > ~[?:1.8.0_60] >> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_60] >> at > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto____7720.invoke(worker.clj:515) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.AFn.applyToHelper(AFn.java:178) > [clojure-1.6.0.jar:?] >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] >> at clojure.core$apply.invoke(core.clj:624) [clojure-1.6.0.jar:?] >> at > backtype.storm.daemon.worker$fn__7719$mk_worker__7805.doInvoke(worker.clj:491) > > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:?] >> at backtype.storm.daemon.worker$_main.invoke(worker.clj:641) > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> at clojure.lang.AFn.applyToHelper(AFn.java:165) > [clojure-1.6.0.jar:?] >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] >> at backtype.storm.daemon.worker.main(Unknown Source) > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > -- > > Jon
signature.asc
Description: Message signed with OpenPGP using GPGMail
