What type of grouping are you using to distribute the tuples to the bolts? Is it possible that it is a field grouping where there is a skew in the values used to group?
On Thu, Oct 30, 2014 at 1:47 AM, Yiorgos Panayiotakis < [email protected]> wrote: > We have a topology which consists of a KafkaSpout and a persistence bolt > which is used > to write in a memory database and then forward tuples to other bolts. The > issue we are facing is that the topology does not scale as expected. > Specifically when we scale our topology using 2 machines - 2 workers(8 > cores - 16 parallesism units) everything seems to scale normally. We use 2 > KafkaSpouts which distribute tuples to 2 persistence bolts almost evenly > (we use field grouping based on device id between spouts and bolts). > But our problem starts when we scale to 4 machines - 4 workers. Then even > though the 4 KafkaSpouts emit almost the same number of tuples those are > not distributed to the 4 persistence bolts. In fact half of those emitted > tuples result in only 1 bolt and another bolt does not get any tuples (the > 2 other get a small number o f tuples). > My question is why this is happening ? It is getting much worse if we > increase the number of executors for the persistence bolts (e.g. 8 > executors result only half of the bolts get any tuples), but at the same > point when we increase the number of tasks to a large number 64 for example > then every bolt gets some tuples. Is this the normal behavior ? How can we > resolve this issue ? >
