If possible can you post some logs from supervisor.log. Interested in looking at the log when your supervisor starts.
-Harsha On Thu, Aug 28, 2014, at 07:29 AM, Vikas Agarwal wrote: Yes, I am through it. I have killed the processes created by main supervisor processes for 6700 and 6701 ports and then started process for one of these ports. After that I faced issues due to multiple versions of same library in storm lib e.g. netty and servlet-api After that I faced this stack overflow issue. Now, I am even able to fix it. Multiple slf4j-log4j implementations was the issue behind stack overflow. Now, I am back to the same state where the process just don't start. Now running the worker command manually is even not showing any log except this: JMXetricAgent instrumented JVM, see [1]https://github.com/ganglia/jmxetric Aug 28, 2014 10:28:39 AM info.ganglia.gmetric4j.GMonitor start INFO: Setting up 1 samplers And then process get killed. On Thu, Aug 28, 2014 at 7:22 PM, Harsha <[2][email protected]> wrote: Vikas, Are you able to get past this error "Running the command manually on console causes "Address already in use" error for supervisor ports (6700,6701)". Did you check if there are any processes running on that port. -Harsha On Thu, Aug 28, 2014, at 01:58 AM, Vikas Agarwal wrote: I am getting following error when trying to run the command for worker directly on console Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "main-SendThread(hdp.ambari:2181)" Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "Thread-2" Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "Thread-12-<bolt1>" Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "Thread-10-<bolt2>" Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "Thread-8-<bolt3>" Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "Thread-14-<spout>" Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "Thread-14-feed-stream-SendThread(localhost:2181)" Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "Thread-14-feed-stream-SendThread(localhost:2181)" Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "Thread-14-feed-stream-SendThread(hdp.ambari:2181)" As one of the possible bug situations, I looked for multiple netty jars as suggested in other mail thread, it didn't work. Can anyone help me out where should I look next to resolve the issue. On Tue, Aug 26, 2014 at 2:20 PM, Vikas Agarwal <[3][email protected]> wrote: However, now my topology is failing to start worker process again. :( This time is not showing me any good clue to resolve it. Running the command manually on console causes "Address already in use" error for supervisor ports (6700,6701). So, it is not letting me move forward to see what actually the error is while running the worker. On Mon, Aug 25, 2014 at 9:00 PM, Vikas Agarwal <[4][email protected]> wrote: Yes, I was able to see the topology in Storm UI and nothing was logged into worker logs. However, as I mentioned, I am able to resolve it by finding an hint in supervisor.log file this time. On Mon, Aug 25, 2014 at 8:58 PM, Georgy Abraham <[5][email protected]> wrote: Are you able to see the topology in storm UI or with storm list command ?? And worker mentioned in the UI doesn't have any log ?? __________________________________________________________ From: Vikas Agarwal Sent: 25-08-2014 PM 05:25 To: [6][email protected] Subject: Storm not processing topology without logs Hi, I have started to explore the Storm for distributed processing for our use case which we were earlier fulfilling by JMS based MQ system. Topology worked after some efforts. It has one spout (KafkaSpout from kafka-storm project) and 3 bolts. First bolt sets context for other two bolts which in turn do some processing on the tuples and persist the analyzed results in some DB (Mongo, Solr, HBase etc). Recently the topology stopped working. I am able to submit the topology and it does not throw any error in submitting the topology, however, nimbus.log or worker-6701.log files are not showing any progress and eventually topology does not consume any message. I don't have doubt on KafkaSpout because if it was the culprit, at least some initialization logs of spout and bolts should have been there in nimbus.log or worker-xxxx.log. Isn't it? Here is the snippet of nimbus.log after uploading the jar to cluster Uploading file from client to /hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe1 3c706b2ab.jar 2014-08-25 07:07:49 b.s.d.nimbus [INFO] Finished uploading file from client: /hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe1 3c706b2ab.jar 2014-08-25 07:07:49 b.s.d.nimbus [INFO] Received topology submission for aleads with conf {"topology.max.task.parallelism" nil, "topology.acker.executors" nil, "topology.kryo.register" nil, "topology.kryo.decorators" (), "[7]topology.name" "aleads", "[8]storm.id" "aleads-3-1408964869", "modelId" "ut", "topology.workers" 1, "topology.debug" true} 2014-08-25 07:07:50 b.s.d.nimbus [INFO] Activating aleads: aleads-3-1408964869 2014-08-25 07:07:50 b.s.s.EvenScheduler [INFO] Available slots: (["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6700]) 2014-08-25 07:07:50 b.s.d.nimbus [INFO] Setting new assignment for topology id aleads-3-1408964869: #backtype.storm.daemon.common.Assignment{:master-code-dir "/hadoop/storm/nimbus/stormdist/aleads-3-1408964869", :node->host {"e56c2cc7-d35a-4355-9906-506618ff70c5" "hdp.ambari"}, :executor->node+port {[2 2] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [3 3] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [4 4] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [5 5] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [6 6] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [7 7] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [8 8] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [9 9] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [1 1] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701]}, :executor->start-time-secs {[1 1] 1408964870, [9 9] 1408964870, [8 8] 1408964870, [7 7] 1408964870, [6 6] 1408964870, [5 5] 1408964870, [4 4] 1408964870, [3 3] 1408964870, [2 2] 1408964870}} Can anyone guess what I have done wrong and why Storm is not giving any error log anywhere. Storm version is 0.9.1.2.1.3.0-563 (Installed via HortonWorks) Kafka version is 2.10-0.8.1.1 Storm-Kafka version 0.9.2-incubating -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters [9]http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 [10]+1 (408) 988-2000 Work [11]+1 (408) 716-2726 Fax -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters [12]http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 [13]+1 (408) 988-2000 Work [14]+1 (408) 716-2726 Fax -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters [15]http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 [16]+1 (408) 988-2000 Work [17]+1 (408) 716-2726 Fax -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters [18]http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 [19]+1 (408) 988-2000 Work [20]+1 (408) 716-2726 Fax -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters [21]http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax References 1. https://github.com/ganglia/jmxetric 2. mailto:[email protected] 3. mailto:[email protected] 4. mailto:[email protected] 5. mailto:[email protected] 6. mailto:[email protected] 7. http://topology.name/ 8. http://storm.id/ 9. http://www.infoobjects.com/ 10. tel:%2B1%20%28408%29%20988-2000 11. tel:%2B1%20%28408%29%20716-2726 12. http://www.infoobjects.com/ 13. tel:%2B1%20%28408%29%20988-2000 14. tel:%2B1%20%28408%29%20716-2726 15. http://www.infoobjects.com/ 16. tel:%2B1%20%28408%29%20988-2000 17. tel:%2B1%20%28408%29%20716-2726 18. http://www.infoobjects.com/ 19. tel:%2B1%20%28408%29%20988-2000 20. tel:%2B1%20%28408%29%20716-2726 21. http://www.infoobjects.com/
