Hi, Few questions on your issue:
1. As soon as you start the topology, is that the bolt execution is not started forever ? or is it like after processing few tuples, bolt execution has stuck. Can you give clear picture on this. 2. You said that the behavior is seen on one of the worker process. So can is that on the other worker process, bolt is executed without any issues ? 3. What is the source from which spout reads the input ? Also, it would be great if you could provide nimbus/supervisor and worker logs ? On Mon, May 12, 2014 at 6:57 AM, Srinath C <[email protected]> wrote: > Hi, > I'm facing a strange issue running a topology on version > 0.9.1-incubating with Netty as transport. > The topology has two worker processes on the same worker machine. > > To summarize the behavior, on one of the worker processes: > - one of the bolts are not getting executed: The bolt has multiple > executors of the same bolt but none of them are executed > - the spouts in the same worker process are trying to emit tuples to > the bolt but still the bolt is not executed > - after a while the spout itself is not executed (nextTuple is not > called) > > My suspicion is that due to some internal buffers getting filled up > the topology.max.spout.pending limit is hit and storm is no longer invoking > the spout. The topology remains hung like this for a long time and probably > for ever. > From the jstack output, I could figure out that there were 5 threads > lesser in the affected process than a normal process. The thread were > having a stack as below: > > Thread 5986: (state = BLOCKED) > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) > @bci=68, line=2082 (Compiled frame) > - > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() > @bci=122, line=1090 (Compiled frame) > - > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() > @bci=1, line=807 (Interpreted frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 > (Interpreted frame) > - > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=26, line=1130 (Interpreted frame) > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame) > > Has anyone seen such an issue? Any idea if I can confirm my suspicion > of internal buffers getting filled up? What else can I collect from the > processes for troubleshooting? > > Thanks, > Srinath. > >
