There are 4 supervisor machines in cluster and 4 spout are tasks are running . A usual tuple takes like 0,5 second to process and duration rises above 15 seconds when queue is full. I will try to increase the number to 500 as you have suggested.
Thanks, 12 May 2015 Sal, 16:05 tarihinde, Nathan Leung <[email protected]> şunu yazdı: > When the spout output queue is big then you will see the total processing > time increase because it includes the time the tuple spends in the queue. > How many spout tasks do you have? Your max pending seems low, and when it's > too low your cluster will be starved for data. Try increasing it to a few > hundred or one thousand. > On May 12, 2015 7:23 AM, "Kutlu Araslı" <[email protected]> wrote: > >> Hi everyone, >> >> Our topology consumes tuples from a Kestrel MQ and runs a series of bolts >> to process items including some db connections. Storm version is 0.8.3 and >> supervisors are run on VMs. >> When number of tuples increases in queue, we observe that, a single tuple >> execution time also rise dramatically in paralel which ends up with a >> throttle behaviour. >> In the meantime CPU and memory usage looks comfortable.From database >> point, we have not observed a problem so far under stress. >> Is there any configuration trick or an advice for handling such a load? >> There is already a limit on MAX_SPOUT_PENDING as 32. >> >> Thanks, >> >> >>
