When the spout output queue is big then you will see the total processing time increase because it includes the time the tuple spends in the queue. How many spout tasks do you have? Your max pending seems low, and when it's too low your cluster will be starved for data. Try increasing it to a few hundred or one thousand. On May 12, 2015 7:23 AM, "Kutlu Araslı" <[email protected]> wrote:
> Hi everyone, > > Our topology consumes tuples from a Kestrel MQ and runs a series of bolts > to process items including some db connections. Storm version is 0.8.3 and > supervisors are run on VMs. > When number of tuples increases in queue, we observe that, a single tuple > execution time also rise dramatically in paralel which ends up with a > throttle behaviour. > In the meantime CPU and memory usage looks comfortable.From database > point, we have not observed a problem so far under stress. > Is there any configuration trick or an advice for handling such a load? > There is already a limit on MAX_SPOUT_PENDING as 32. > > Thanks, > > >
