I added a 60s wait and still saw the issue but less frequently. In the middle of upgrading to master to get the 1.0.x changes.
Jon On Wed, Nov 2, 2016, 19:41 James Sirota <[email protected]> wrote: > > Resurrecting this thread. Were we able to come to a resolution on this? > Or is it still causing problems? > > Thanks, > James > > > 05.10.2016, 12:55, "[email protected]" <[email protected]>: > > Right, I will try that. I guess my main question, though, is why would > > that need more time? Is it a function of CPU, memory, disk, or otherwise? > > Obviously if the timeout is the issue it's probably related to what I'm > > sending into the cluster and how resources are allocated in relation to > the > > workload but I plan to scale this up roughly 10x and don't want to end up > > with a 600 second timeout. > > > > Jon > > > > On Wed, Oct 5, 2016, 15:34 Otto Fowler <[email protected]> wrote: > > > > What you would have to do to try this is to edit > > /usr/local/monit/stop_bro_topology.sh and add in a -w parameter. > > to read: > > > > storm kill bro -w 60 > > > > save that and then try to stop if from monit again. > > > > If that works, then we will know. > > > > On October 5, 2016 at 15:18:43, [email protected] ([email protected]) > wrote: > > > > topology.message.timeout.secs is set to 30 according to ambari. > > > > What you requested is essentially what I did before, except instead of > > using `storm kill` I used monit to kill all of the topologies, and I > waited > > multiple minutes before stopping nimbus/supervisor/drpc, then waited a > > couple more minutes before grabbing the prior metrics/processes. I've > seen > > this multiple times across the past week and each time I see it I shut > > everything down, do a `kill -9` on the storm processes left behind, make > > sure no processes are running as the storm user on my entire cluster, > then > > bring things up in ambari, then monit once ambari says > > nimbus/supervisor/drpc is up. > > > > Jon > > > > On Wed, Oct 5, 2016 at 3:09 PM P. Taylor Goetz <[email protected]> > wrote: > > > >> To build on what Otto said: > >> > >> What’s your topology.message.timeout.secs configuration value? The > > > > default > >> is 30. > >> > >> If you issue the `storm kill` command without the -w option, Storm will > >> first deactivate the topology, then wait topology.message.timeout.secs > >> before actually shutting down the topology. > >> > >> Also, if you have topologies deployed, and stop ninmubs, supervisors, > > > > etc. > >> the workers may continue to run (depending on how you have monit > setup). > >> > >> Can you try the following: > >> > >> 1. Kill the topology and wait topology.message.timeout.sec so Storm > >> properly shuts down the topology. > >> 2. Stop nimbus/supervisor/etc. > >> 3. Check to see if you still have threads left behind. > >> > >> -Taylor > >> > >> > On Oct 5, 2016, at 2:31 PM, Otto Fowler <[email protected]> > > > > wrote: > >> > > >> > The shutdown topology scripts only do a storm kill <item>. > >> > They probably need to -w WAITTIME. > >> > > >> > > >> > -- > >> > > >> > Sent with Airmail > >> > > >> > On October 5, 2016 at 13:53:33, [email protected] ([email protected]) > >> wrote: > >> > > >> > Hi All, > >> > > >> > While tuning Metron I have been doing a large amount of storm > topology > >> > restarts, as I tune the parallelism and memory use of bolts. I have > >> > noticed multiple times that some threads will be left behind on > > > > restart, > >> > which eventually causes all topologies to fail to start due to > >> > "java.lang.OutOfMemoryError: unable to create new native thread". I > am > >> > only using the bro, indexing, and enrichments topologies right now, > and > > > > I > >> > have seen all of them leave hanging processes behind. Has anybody > else > >> > seen anything like this? > >> > > >> > I'm exclusively using monit to do topology restarts, and Ambari to > >> restart > >> > supervisors/nimbus/UI/DRPC processes. When it is currently > registering > > > > as > >> > stopped in both monit and Ambari, I see the following threads left > >> behind: > >> >> [jzeolla@node1 storm]$ ps -austorm | wc -l > >> >> 6 > >> >> [jzeolla@node2 storm]$ ps -austorm | wc -l > >> >> 12 > >> >> [jzeolla@node3 storm]$ ps -austorm | wc -l > >> >> 9 > >> >> [jzeolla@node4 storm]$ ps -austorm | wc -l > >> >> 11 > >> >> [jzeolla@node5 storm]$ ps -austorm | wc -l > >> >> 9 > >> > > >> > Some sample processes: > >> >> storm 14257 0.1 0.2 6279184 293912 ? Sl Oct03 3:44 > >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer > >> > -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > >> > -Dstorm.log.dir=/var/log/storm > >> > > > > > > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > >> > > >> > -Dstorm.conf.file= -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > >> > > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf > > > >> > > >> > -Xmx128m -Dlogfile.name=logviewer.log > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml > >> > backtype.storm.daemon.logviewer > >> >> storm 32072 0.0 0.1 37337056 177112 ? Sl Oct02 2:02 > >> > /usr/jdk64/jdk1.8.0_60/bin/java -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > >> > > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.id=bro-13-1475456606 > >> > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 > >> > -Dstorm.log.dir=/var/log/storm > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > > > > -Xmx20G > >> > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > > >> > > >> > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > >> > > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > >> -Dstorm.options= > >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > -Dstorm.id=bro-13-1475456606 > >> > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 > -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > >> > > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > backtype.storm.daemon.worker bro-13-1475456606 > >> > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6704 > >> > f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 > >> >> storm 32127 121 7.4 32422464 9799588 ? Sl Oct02 4337:42 > >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > >> > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > > >> > > >> > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > >> > > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > >> -Dstorm.options= > >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > -Dstorm.id=bro-13-1475456606 > >> > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 > -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > >> > > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > backtype.storm.daemon.worker bro-13-1475456606 > >> > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6704 > >> > f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 > >> >> storm 47882 0.0 0.1 37338556 188920 ? Sl Oct04 0:47 > >> > /usr/jdk64/jdk1.8.0_60/bin/java -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > >> > > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > -Dlogfile.name=indexing-16-1475589543-worker-6700.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > > > > -Dstorm.id=indexing-16-1475589543 > >> > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 > >> > -Dstorm.log.dir=/var/log/storm > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > -Xmx2G > >> > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > > >> > > >> > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > >> > > >> > -Dlogfile.name=indexing-16-1475589543-worker-6700.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > >> -Dstorm.options= > >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > -Dstorm.id=indexing-16-1475589543 > >> > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 > -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > >> > > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > backtype.storm.daemon.worker indexing-16-1475589543 > >> > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6700 > >> > 5e17c5c3-e6ab-4b48-a48a-dc540cda7087 > >> >> storm 47978 4.7 0.5 19996036 740372 ? Sl Oct04 65:04 > >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx2G > >> > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > > >> > > >> > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > >> > > >> > -Dlogfile.name=indexing-16-1475589543-worker-6700.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > >> -Dstorm.options= > >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > -Dstorm.id=indexing-16-1475589543 > >> > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 > -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > > > >> > > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > > > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > backtype.storm.daemon.worker indexing-16-1475589543 > >> > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6700 > >> > 5e17c5c3-e6ab-4b48-a48a-dc540cda7087 > >> >> storm 21301 0.1 0.2 6347824 298632 ? Sl Oct03 3:44 > >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer > >> > -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > >> > -Dstorm.log.dir=/var/log/storm > >> > > > > > > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > >> > > >> > -Dstorm.conf.file= -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > >> > > >> > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf > > > >> > > >> > -Xmx128m -Dlogfile.name=logviewer.log > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml > >> > backtype.storm.daemon.logviewer > >> >> storm 14046 102 19.3 42125944 25539856 ? Sl Oct03 2655:13 > >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > >> > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > > >> > > >> > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > >> > > >> > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > >> -Dstorm.options= > >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > -Dstorm.id=enrichment-12-1475456272 > >> > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 > -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > >> > > >> > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > backtype.storm.daemon.worker enrichment-12-1475456272 > >> > eff7f971-ebe2-4f53-beab-99b061bfde10 6700 > >> > 5d586063-e405-4a67-a644-53a0ac943ab0 > >> >> storm 6544 0.0 0.1 37339132 180872 ? Sl Oct02 3:22 > >> > /usr/jdk64/jdk1.8.0_60/bin/java -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > >> > > >> > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.id=bro-13-1475456606 > >> > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 > >> > -Dstorm.log.dir=/var/log/storm > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > > > > -Xmx20G > >> > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > > >> > > >> > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > >> > > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > >> -Dstorm.options= > >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > -Dstorm.id=bro-13-1475456606 > >> > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 > -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > >> > > >> > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > backtype.storm.daemon.worker bro-13-1475456606 > >> > eff7f971-ebe2-4f53-beab-99b061bfde10 6704 > >> > f917291c-bd1e-4a9c-a52d-b9c5fef0567e > >> >> storm 6592 192 9.8 34986568 12987704 ? Sl Oct02 6866:30 > >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G > >> > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > > > >> > > >> > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > >> > > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > >> -Dstorm.options= > >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > -Dstorm.id=bro-13-1475456606 > >> > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 > -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > >> > > >> > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > backtype.storm.daemon.worker bro-13-1475456606 > >> > eff7f971-ebe2-4f53-beab-99b061bfde10 6704 > >> > f917291c-bd1e-4a9c-a52d-b9c5fef0567e > >> >> storm 13998 0.0 0.1 37339104 184140 ? Sl Oct03 1:57 > >> > /usr/jdk64/jdk1.8.0_60/bin/java -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > >> > > >> > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > >> -Dstorm.id=enrichment-12-1475456272 > >> > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 > >> > -Dstorm.log.dir=/var/log/storm > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > > > > -Xmx20G > >> > > > > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > > > >> > > >> > > > > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > > >> > > >> > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log > >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= > >> -Dstorm.options= > >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 > >> > > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml > >> > > > > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > > > >> > > >> > -Dstorm.id=enrichment-12-1475456272 > >> > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 > -cp > >> > > > > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > > > >> > > >> > 2.4.0. > >> > > > > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > > > >> > > >> > > > > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > > > >> > > >> > backtype.storm.daemon.worker enrichment-12-1475456272 > >> > eff7f971-ebe2-4f53-beab-99b061bfde10 6700 > >> > 5d586063-e405-4a67-a644-53a0ac943ab0 > >> > > >> > > >> > Logs showing the inability to start a topology: > >> >> 2016-10-04 10:10:57.787 b.s.m.n.Client [INFO] creating Netty Client, > >> > connecting to pgh-dat-t53.iso.local.cmu.edu:6702, bufferSize: > 5242880 > >> >> 2016-10-04 10:10:57.787 o.a.s.c.r.ExponentialBackoffRetry [WARN] > >> > maxRetries too large (30). Pinning to 29 > >> >> 2016-10-04 10:10:57.787 b.s.u.StormBoundedExponentialBackoffRetry > > > > [INFO] > >> > The baseSleepTimeMs [100] the maxSleepTimeMs [1000] the maxRetries > [30] > >> >> 2016-10-04 10:10:57.787 b.s.m.n.Client [INFO] connection attempt 1 > to > >> > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6702 > scheduled > >> to > >> > run in 0 ms > >> >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established > > > > to > >> > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6703 > >> >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established > > > > to > >> > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6702 > >> >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established > > > > to > >> > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6701 > >> >> 2016-10-04 10:10:58.030 b.s.d.executor [INFO] Loading executor > >> > enrichmentJoinBolt:[2 2] > >> >> 2016-10-04 10:10:58.064 b.s.d.executor [INFO] Loaded executor tasks > >> > enrichmentJoinBolt:[2 2] > >> >> 2016-10-04 10:10:58.071 b.s.d.executor [INFO] Finished loading > > > > executor > >> > enrichmentJoinBolt:[2 2] > >> >> 2016-10-04 10:10:58.081 b.s.d.executor [INFO] Loading executor > >> > simpleHBaseThreatIntelBolt:[22 22] > >> >> 2016-10-04 10:10:58.095 b.s.d.executor [INFO] Loaded executor tasks > >> > simpleHBaseThreatIntelBolt:[22 22] > >> >> 2016-10-04 10:10:58.097 b.s.d.executor [INFO] Finished loading > > > > executor > >> > simpleHBaseThreatIntelBolt:[22 22] > >> >> 2016-10-04 10:10:58.104 b.s.d.executor [INFO] Loading executor > >> > kafkaSpout:[18 18] > >> >> 2016-10-04 10:10:58.112 b.s.d.executor [INFO] Loaded executor tasks > >> > kafkaSpout:[18 18] > >> >> 2016-10-04 10:10:58.118 b.s.d.executor [INFO] Finished loading > > > > executor > >> > kafkaSpout:[18 18] > >> >> 2016-10-04 10:10:58.125 b.s.d.executor [INFO] Loading executor > >> > geoEnrichmentBolt:[6 6] > >> >> 2016-10-04 10:10:58.135 b.s.d.executor [INFO] Loaded executor tasks > >> > geoEnrichmentBolt:[6 6] > >> >> 2016-10-04 10:10:58.137 b.s.d.executor [ERROR] > >> >> java.lang.OutOfMemoryError: unable to create new native thread > >> >> at java.lang.Thread.start0(Native Method) ~[?:1.8.0_60] > >> >> at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_60] > >> >> at backtype.storm.util$async_loop.doInvoke(util.clj:491) > >> > ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.RestFn.invoke(RestFn.java:559) > >> > ~[clojure-1.6.0.jar:?] > >> >> at backtype.storm.daemon.executor$fn__7245.invoke(executor.clj:820) > >> > ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.MultiFn.invoke(MultiFn.java:236) > >> > ~[clojure-1.6.0.jar:?] > >> >> at > >> > > > > > > backtype.storm.daemon.executor$mk_executor$fn__7116.invoke(executor.clj:351) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at > >> > backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:350) > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at > >> > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721$iter__7726__7730$fn__7731.invoke(worker.clj:541) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.6.0.jar:?] > >> >> at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.6.0.jar:?] > >> >> at clojure.lang.Cons.next(Cons.java:39) [clojure-1.6.0.jar:?] > >> >> at clojure.lang.RT.next(RT.java:598) [clojure-1.6.0.jar:?] > >> >> at clojure.core$next.invoke(core.clj:64) [clojure-1.6.0.jar:?] > >> >> at clojure.core$dorun.invoke(core.clj:2856) [clojure-1.6.0.jar:?] > >> >> at clojure.core$doall.invoke(core.clj:2871) [clojure-1.6.0.jar:?] > >> >> at > >> > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721.run(worker.clj:541) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at java.security.AccessController.doPrivileged(Native Method) > >> > ~[?:1.8.0_60] > >> >> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_60] > >> >> at > >> > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto____7720.invoke(worker.clj:515) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.AFn.applyToHelper(AFn.java:178) > >> > [clojure-1.6.0.jar:?] > >> >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] > >> >> at clojure.core$apply.invoke(core.clj:624) [clojure-1.6.0.jar:?] > >> >> at > >> > > > > > > backtype.storm.daemon.worker$fn__7719$mk_worker__7805.doInvoke(worker.clj:491) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:?] > >> >> at backtype.storm.daemon.worker$_main.invoke(worker.clj:641) > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.AFn.applyToHelper(AFn.java:165) > >> > [clojure-1.6.0.jar:?] > >> >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] > >> >> at backtype.storm.daemon.worker.main(Unknown Source) > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> 2016-10-04 10:10:58.169 b.s.util [ERROR] Halting process: ("Worker > >> died") > >> >> java.lang.RuntimeException: ("Worker died") > >> >> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:332) > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?] > >> >> at > >> > backtype.storm.daemon.worker$fn__7818$fn__7819.invoke(worker.clj:636) > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at > >> > > > > > > backtype.storm.daemon.executor$mk_executor_data$fn__7078$fn__7079.invoke(executor.clj:256) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at > >> > > > > > > backtype.storm.daemon.executor$mk_executor$fn__7116.invoke(executor.clj:350) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at > >> > backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:350) > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at > >> > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721$iter__7726__7730$fn__7731.invoke(worker.clj:541) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.6.0.jar:?] > >> >> at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.6.0.jar:?] > >> >> at clojure.lang.Cons.next(Cons.java:39) [clojure-1.6.0.jar:?] > >> >> at clojure.lang.RT.next(RT.java:598) [clojure-1.6.0.jar:?] > >> >> at clojure.core$next.invoke(core.clj:64) [clojure-1.6.0.jar:?] > >> >> at clojure.core$dorun.invoke(core.clj:2856) [clojure-1.6.0.jar:?] > >> >> at clojure.core$doall.invoke(core.clj:2871) [clojure-1.6.0.jar:?] > >> >> at > >> > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721.run(worker.clj:541) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at java.security.AccessController.doPrivileged(Native Method) > >> > ~[?:1.8.0_60] > >> >> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_60] > >> >> at > >> > > > > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto____7720.invoke(worker.clj:515) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.AFn.applyToHelper(AFn.java:178) > >> > [clojure-1.6.0.jar:?] > >> >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] > >> >> at clojure.core$apply.invoke(core.clj:624) [clojure-1.6.0.jar:?] > >> >> at > >> > > > > > > backtype.storm.daemon.worker$fn__7719$mk_worker__7805.doInvoke(worker.clj:491) > > > >> > > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:?] > >> >> at backtype.storm.daemon.worker$_main.invoke(worker.clj:641) > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> >> at clojure.lang.AFn.applyToHelper(AFn.java:165) > >> > [clojure-1.6.0.jar:?] > >> >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] > >> >> at backtype.storm.daemon.worker.main(Unknown Source) > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] > >> > -- > >> > > >> > Jon > >> > >> -- > > > > Jon > > > > -- > > > > Jon > > ------------------- > Thank you, > > James Sirota > PPMC- Apache Metron (Incubating) > jsirota AT apache DOT org > -- Jon
