Hi, Maybe I asked too soon.
http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/ clearly says that the number of tasks is fixed for lifetime of the topology. Number of executors == number of tasks by default. Hence, if the number of executors is already at maximum level, it cannot go further. In the comments section, it further says: "Configuring multiple tasks per executor (thread) gives you the flexibility to expand the topology through the "storm rebalance" command in the future without taking the topology offline. In other words it is not a performance reason." So, it appears the expectation is to overprovision the number of tasks, start with minimal number of executors, and then grow executors to achieve parallelism as workload increases. Is this right ? If yes, overall, this is somewhat operationally difficult than expanding on need. Is this a valid use case for Storm to support ? Any plans for this in future ? Thanks hemanth On Mon, Jun 22, 2015 at 5:34 PM, John Yost <[email protected]> wrote: > This is an excellent question, need this info for myself as well. > > --John > > On Mon, Jun 22, 2015 at 7:40 AM, Hemanth Yamijala <[email protected]> > wrote: > >> Hi, >> >> I was testing the rebalance functionality on Storm 0.9.4. >> >> storm rebalance <name> -w 10 -n 2 >> - Works as expected. It increased the number of workers to 2. >> >> storm rebalance <name> -w 10 -n 2 -e <bolt-name>=20 >> - Works only for increasing the number of workers, but did *not* change >> the number of executors of <bolt-name>. >> >> I came across this question on StackOverflow: >> http://stackoverflow.com/questions/18716780/storm-v0-8-2-rebalance-command-not-updating-the-number-of-executors-for-a-bolt >> >> >> which seems to indicate that the number of executors is bounded by the >> number of tasks, and if number of executors = number of tasks (which it is >> in my case), we can't increase this further. >> >> Is this right ? Is there a way we can increase the number of executors at >> run time without restarting the topology (along with increasing number of >> workers) ? >> >> Thanks >> Hemanth >> > >
