What you are asking is very specific to the code in the bolts. A storm topology is a division of labour. Given 2 instances of a bolt. Both bolts take their input from the same sender. Their outputs are combined into Bolt 3. Bolt 1 must contact a database, write a log, interact with a web api. Bolt 2 must add a static field to the tuple and pass it on. It would make complete sense that the execution time in Bolt 1 is much higher, and therefore its capacity much lower and therefore, Bolt3 may end up 'waiting' on Bolt1. In this case, the Storm thing to do would be to increase the number of instances of the Bolt 1 running in the topology.
https://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html http://stackoverflow.com/questions/27756085/configuring-parallelism-in-storm From: Aditya Rajan <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: 2015,Tuesday, May 5 at 10:07 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Unbalanced Executors Hey Jeff, Turns out that all the executors receive the same number of messages however when one executor receives more and finishes faster, it has to wait till the others finish as well, or something to that effect. Is there anyway I can group them to keep all the executors at maximum capacity On May 5, 2015 8:07 PM, "Jeff Maass" <[email protected]<mailto:[email protected]>> wrote: I don't know about "normal", but expected would be as per the documentation. See Stream Groupings : https://storm.apache.org/documentation/Concepts.html You will find the specifics of your situation in the definition of your topology. For Shuffle / all / none Grouping, what you described would be unexpected. For Local or Shuffle Grouping, what you described would be expected, but, only if the sending and receiving bolt are in the same executor. For Filed / partial key / global / direct - whether or not what you described would be expected depends upon your topology's application. From: Aditya Rajan <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: 2015,Tuesday, May 5 at 09:26 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Unbalanced Executors Hey Everyone, My topology seems to be running fine. However when I look at the executors for each bolt, some are always at 40-50% capacity and others are always at 90-100% capacity. Is this normal? Is there any way to manually smoothen this?
