I did try increasing the executor buffers to 16384 from default 1024, but did not get any benefits. It was the same problem. In any case I would assume the spout should wait for 30 secs (message timeout) to fail them. But as highlighted in below logs the spout immediately fails it which is surprising.
I monitored the jstack, and almost all bolts are in TIMED_WAITING state waiting on the data to be available in disruptor queue. Any more ideas on this ? Also is there a way to monitor the buffers and identify if we are saturating them because even with a vey small spout pending i.e 50, I see that it runs fine initially and then starts slowing down till some bolts completely stop processing. Thanks Indra On 21 Jun 2014 06:12, "Srinath C" <[email protected]> wrote: > Is this happening when there are a lot of tuples emitted? I suspect its > because of buffers getting filled. > Check the "capacity" of the bolt to which these tuples are getting > transferred to (in the metrics page on storm ui). > Once you confirm that, try increasing the buffers for the executors. > > > On Fri, Jun 20, 2014 at 10:07 PM, Indra Nath Bardhan < > [email protected]> wrote: > >> Hi, >> >> We are seeing a strange use-case, where the spout fails the tuple >> immediately after emitting it. We have set the >> topology.message.timeout.secs intially to 30 secs and extended to 60 secs >> as well. >> >> *Below is a log snapshot - You can see the spout failing the tuple >> immediately after emitting with a reason as TIMEOUT* >> >> >> 2014-06-20 16:04:29 [INFO] b.s.d.executor:0 thd=Thread-75-MyFeedReceiver >> tplg= cmpn= trcid= msg= TRANSFERING tuple TASK: 2800 TUPLE: source: >> MyFeedReceiver:15, stream: MyFeederStream, id: >> {-3058727628838667563=3004157440983613205}, [null, >> com.yahoo.boson.base.Message@3c039b2a, BOSON] >> >> 2014-06-20 16:04:29 [INFO] b.s.d.executor:0 thd=Thread-75-MyFeedReceiver >> tplg= cmpn= trcid= msg= TRANSFERING tuple TASK: 30 TUPLE: source: >> MyFeedReceiver:15, stream: __ack_init, id: {}, [-3058727628838667563 >> 3004157440983613205 15] >> >> 2014-06-20 16:04:29 [INFO] b.s.d.executor:0 thd=Thread-75-MyFeedReceiver >> tplg= cmpn= trcid= msg= SPOUT Failing -3058727628838667563: {:stream >> "MyFeederStream", :values #<Values [null, >> com.yahoo.boson.base.Message@3c039b2a, BOSON]>} REASON: TIMEOUT MSG-ID: >> 5ba4b813-43cb-4c25-b6f1-faba4a48a759 >> >> >> Note that this is not the behaviour for all tuples, but only few. Can you >> please let us know what could be a reason for this ? >> >> >> Thanks, >> Indra >> >> >> >
