If your tuples take that long 500 may be too high (depends on your parallelism) but if you are seeing under utilization then 32 is probably too low. You can try different settings to find what works best for your application. On May 12, 2015 9:31 AM, "Kutlu Araslı" <[email protected]> wrote:
> There are 4 supervisor machines in cluster and 4 spout are tasks are > running . A usual tuple takes like 0,5 second to process and duration rises > above 15 seconds when queue is full. > I will try to increase the number to 500 as you have suggested. > > Thanks, > > 12 May 2015 Sal, 16:05 tarihinde, Nathan Leung <[email protected]> şunu > yazdı: > >> When the spout output queue is big then you will see the total processing >> time increase because it includes the time the tuple spends in the queue. >> How many spout tasks do you have? Your max pending seems low, and when it's >> too low your cluster will be starved for data. Try increasing it to a few >> hundred or one thousand. >> On May 12, 2015 7:23 AM, "Kutlu Araslı" <[email protected]> wrote: >> >>> Hi everyone, >>> >>> Our topology consumes tuples from a Kestrel MQ and runs a series of >>> bolts to process items including some db connections. Storm version is >>> 0.8.3 and supervisors are run on VMs. >>> When number of tuples increases in queue, we observe that, a single >>> tuple execution time also rise dramatically in paralel which ends up with >>> a throttle behaviour. >>> In the meantime CPU and memory usage looks comfortable.From database >>> point, we have not observed a problem so far under stress. >>> Is there any configuration trick or an advice for handling such a load? >>> There is already a limit on MAX_SPOUT_PENDING as 32. >>> >>> Thanks, >>> >>> >>>
