To elaborate I would consider something like 50 or 60 to avoid tuple timeouts, unless you have increased that above the default. On May 12, 2015 9:53 AM, "Nathan Leung" <[email protected]> wrote:
> If your tuples take that long 500 may be too high (depends on your > parallelism) but if you are seeing under utilization then 32 is probably > too low. You can try different settings to find what works best for your > application. > On May 12, 2015 9:31 AM, "Kutlu Araslı" <[email protected]> wrote: > >> There are 4 supervisor machines in cluster and 4 spout are tasks are >> running . A usual tuple takes like 0,5 second to process and duration rises >> above 15 seconds when queue is full. >> I will try to increase the number to 500 as you have suggested. >> >> Thanks, >> >> 12 May 2015 Sal, 16:05 tarihinde, Nathan Leung <[email protected]> şunu >> yazdı: >> >>> When the spout output queue is big then you will see the total >>> processing time increase because it includes the time the tuple spends in >>> the queue. How many spout tasks do you have? Your max pending seems low, >>> and when it's too low your cluster will be starved for data. Try increasing >>> it to a few hundred or one thousand. >>> On May 12, 2015 7:23 AM, "Kutlu Araslı" <[email protected]> wrote: >>> >>>> Hi everyone, >>>> >>>> Our topology consumes tuples from a Kestrel MQ and runs a series of >>>> bolts to process items including some db connections. Storm version is >>>> 0.8.3 and supervisors are run on VMs. >>>> When number of tuples increases in queue, we observe that, a single >>>> tuple execution time also rise dramatically in paralel which ends up with >>>> a throttle behaviour. >>>> In the meantime CPU and memory usage looks comfortable.From database >>>> point, we have not observed a problem so far under stress. >>>> Is there any configuration trick or an advice for handling such a load? >>>> There is already a limit on MAX_SPOUT_PENDING as 32. >>>> >>>> Thanks, >>>> >>>> >>>>
