>> So, it appears the expectation is to overprovision the number of tasks, >> start with minimal number of executors, and then grow executors to >> achieve parallelism as workload increases. Is this right ?
Yes. This is correct. However, over provisioning the number of tasks does not result in overhead. If you know the number if distinct key, you can use this number (the dop is limited to the number of distinct keys anyway). So just set this number high and you are fine (ie, it should not be difficult in practice). -Matthias On 06/22/2015 03:05 PM, Hemanth Yamijala wrote: > Hi, > > Maybe I asked too soon. > > http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/ > > clearly says that the number of tasks is fixed for lifetime of the > topology. Number of executors == number of tasks by default. Hence, if > the number of executors is already at maximum level, it cannot go > further. In the comments section, it further says: "Configuring > multiple tasks per executor (thread) gives you the flexibility to expand > the topology through the "storm rebalance" command in the future without > taking the topology offline. In other words it is not a performance reason." > > So, it appears the expectation is to overprovision the number of tasks, > start with minimal number of executors, and then grow executors to > achieve parallelism as workload increases. Is this right ? > > If yes, overall, this is somewhat operationally difficult than expanding > on need. Is this a valid use case for Storm to support ? Any plans for > this in future ? > > Thanks > hemanth > > > > On Mon, Jun 22, 2015 at 5:34 PM, John Yost <[email protected] > <mailto:[email protected]>> wrote: > > This is an excellent question, need this info for myself as well. > > --John > > On Mon, Jun 22, 2015 at 7:40 AM, Hemanth Yamijala > <[email protected] <mailto:[email protected]>> wrote: > > Hi, > > I was testing the rebalance functionality on Storm 0.9.4. > > storm rebalance <name> -w 10 -n 2 > - Works as expected. It increased the number of workers to 2. > > storm rebalance <name> -w 10 -n 2 -e <bolt-name>=20 > - Works only for increasing the number of workers, but did *not* > change the number of executors of <bolt-name>. > > I came across this question on > StackOverflow: > http://stackoverflow.com/questions/18716780/storm-v0-8-2-rebalance-command-not-updating-the-number-of-executors-for-a-bolt > > > which seems to indicate that the number of executors is bounded > by the number of tasks, and if number of executors = number of > tasks (which it is in my case), we can't increase this further. > > Is this right ? Is there a way we can increase the number of > executors at run time without restarting the topology (along > with increasing number of workers) ? > > Thanks > Hemanth > > >
signature.asc
Description: OpenPGP digital signature
