Hi,
Workers are restarting very quickly after startup (30s-40s). I am running a
topology on my laptop for testing, so I am guessing tasks are running into some
error. The problem is I don't see any error reported on the log, even though i
have the logging level set to debug. Please see a snippet of the worker log
below.
Note that the bolts are reported to be starting, then there is 37s (23:21:31 -
23:22:08) pause in the log, afterwards the worker seems to restart. Can anyone
guess what might be going on given what's in the log? The second question is
there something one can do to make the log more informative and capture
whatever error that might have happened?
Thanks,Khaled
2014-04-01 23:21:30 b.s.d.executor [INFO] Loading executor IngaBolt_01_003:[2
2]2014-04-01 23:21:31 b.s.d.task [INFO] Emitting: IngaBolt_01_003 __system
["startup"]2014-04-01 23:21:31 b.s.d.executor [INFO] Loaded executor tasks
IngaBolt_01_003:[2 2]2014-04-01 23:21:31 b.s.d.executor [INFO] Finished loading
executor IngaBolt_01_003:[2 2]2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG]
Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null
serverPath:null finished:false header:: 20,3 replyHeader:: 20,74078,0
request:: '/storm/assignments,F response::
s{47,47,1395528580739,1395528580739,0,61,0,0,1,1,74071} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 21,3 replyHeader::
21,74078,0 request:: '/storm/storms,F response::
s{48,48,1395528580741,1395528580741,0,61,0,0,1,1,74070} 2014-04-01 23:21:31
b.s.d.executor [INFO] Preparing bolt IngaBolt_01_003:(2)2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 22,3 replyHeader::
22,74078,0 request:: '/storm/supervisors,F response::
s{49,49,1395528580743,1395528580743,0,107,0,0,1,1,74061} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 23,3 replyHeader::
23,74078,0 request:: '/storm/workerbeats,F response::
s{50,50,1395528580745,1395528580745,0,85,0,0,1,1,74069} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 24,3 replyHeader::
24,74078,0 request:: '/storm/errors,F response::
s{51,51,1395528580747,1395528580747,0,26,0,0,1,0,70999} 2014-04-01 23:21:31
b.s.d.executor [INFO] Loading executor IngaBolt_01_048:[3 3]2014-04-01 23:21:31
b.s.d.task [INFO] Emitting: IngaBolt_01_048 __system ["startup"]2014-04-01
23:21:31 b.s.d.executor [INFO] Loaded executor tasks IngaBolt_01_048:[3
3]2014-04-01 23:21:31 b.s.d.executor [INFO] Finished loading executor
IngaBolt_01_048:[3 3]2014-04-01 23:21:31 b.s.d.executor [INFO] Preparing bolt
IngaBolt_01_048:(3)2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG] Reading reply
sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null
finished:false header:: 25,3 replyHeader:: 25,74078,0 request::
'/storm/assignments,F response::
s{47,47,1395528580739,1395528580739,0,61,0,0,1,1,74071} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 26,3 replyHeader::
26,74078,0 request:: '/storm/storms,F response::
s{48,48,1395528580741,1395528580741,0,61,0,0,1,1,74070} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 27,3 replyHeader::
27,74078,0 request:: '/storm/supervisors,F response::
s{49,49,1395528580743,1395528580743,0,107,0,0,1,1,74061} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 28,3 replyHeader::
28,74078,0 request:: '/storm/workerbeats,F response::
s{50,50,1395528580745,1395528580745,0,85,0,0,1,1,74069} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 29,3 replyHeader::
29,74078,0 request:: '/storm/errors,F response::
s{51,51,1395528580747,1395528580747,0,26,0,0,1,0,70999} 2014-04-01 23:21:31
b.s.d.executor [INFO] Loading executor IngaBolt_01_048:[4 4]2014-04-01 23:21:31
b.s.d.task [INFO] Emitting: IngaBolt_01_048 __system ["startup"]2014-04-01
23:21:31 b.s.d.executor [INFO] Loaded executor tasks IngaBolt_01_048:[4
4]2014-04-01 23:21:31 b.s.d.executor [INFO] Finished loading executor
IngaBolt_01_048:[4 4]2014-04-01 23:21:31 b.s.d.executor [INFO] Preparing bolt
IngaBolt_01_048:(4)2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG] Reading reply
sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null
finished:false header:: 30,3 replyHeader:: 30,74078,0 request::
'/storm/assignments,F response::
s{47,47,1395528580739,1395528580739,0,61,0,0,1,1,74071} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 31,3 replyHeader::
31,74078,0 request:: '/storm/storms,F response::
s{48,48,1395528580741,1395528580741,0,61,0,0,1,1,74070} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 32,3 replyHeader::
32,74078,0 request:: '/storm/supervisors,F response::
s{49,49,1395528580743,1395528580743,0,107,0,0,1,1,74061} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 33,3 replyHeader::
33,74078,0 request:: '/storm/workerbeats,F response::
s{50,50,1395528580745,1395528580745,0,85,0,0,1,1,74069} 2014-04-01 23:21:31
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 34,3 replyHeader::
34,74078,0 request:: '/storm/errors,F response::
s{51,51,1395528580747,1395528580747,0,26,0,0,1,0,70999} 2014-04-01 23:22:08
o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.3.3-1073969,
built on 02/23/2011 22:27 GMT2014-04-01 23:22:08 o.a.z.ZooKeeper [INFO] Client
environment:host.name=172.16.114.112014-04-01 23:22:08 o.a.z.ZooKeeper [INFO]
Client environment:java.version=1.7.0_512014-04-01 23:22:08 o.a.z.ZooKeeper
[INFO] Client environment:java.vendor=Oracle Corporation2014-04-01 23:22:08
o.a.z.ZooKeeper [INFO] Client
environment:java.class.path=/Users/kelmeleegy/install/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/storm-core-0.9.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/asm-4.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/clojure-1.4.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/clout-1.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-codec-1.4.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-exec-1.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-io-1.4.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-lang-2.5.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/compojure-1.1.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/core.incubator-0.1.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/guava-13.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/httpcore-4.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/jetty-6.1.26.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/jline-0.9.94.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/joda-time-2.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/json-simple-1.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/junit-3.8.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/kryo-2.17.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/math.numeric-tower-0.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/minlog-1.2.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/objenesis-1.2.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/servlet-api-2.5-20081211.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/servlet-api-2.5.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/conf:storm-local/supervisor/stormdist/test-1-1396419679/stormjar.jar
