Thanks for your response, Matthias. Will try this out and see how it works for us.
Hemanth On Mon, Jun 22, 2015 at 6:49 PM, Matthias J. Sax < [email protected]> wrote: > >> So, it appears the expectation is to overprovision the number of tasks, > >> start with minimal number of executors, and then grow executors to > >> achieve parallelism as workload increases. Is this right ? > > Yes. This is correct. > > However, over provisioning the number of tasks does not result in > overhead. If you know the number if distinct key, you can use this > number (the dop is limited to the number of distinct keys anyway). So > just set this number high and you are fine (ie, it should not be > difficult in practice). > > -Matthias > > > On 06/22/2015 03:05 PM, Hemanth Yamijala wrote: > > Hi, > > > > Maybe I asked too soon. > > > > > http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/ > > > > clearly says that the number of tasks is fixed for lifetime of the > > topology. Number of executors == number of tasks by default. Hence, if > > the number of executors is already at maximum level, it cannot go > > further. In the comments section, it further says: "Configuring > > multiple tasks per executor (thread) gives you the flexibility to expand > > the topology through the "storm rebalance" command in the future without > > taking the topology offline. In other words it is not a performance > reason." > > > > So, it appears the expectation is to overprovision the number of tasks, > > start with minimal number of executors, and then grow executors to > > achieve parallelism as workload increases. Is this right ? > > > > If yes, overall, this is somewhat operationally difficult than expanding > > on need. Is this a valid use case for Storm to support ? Any plans for > > this in future ? > > > > Thanks > > hemanth > > > > > > > > On Mon, Jun 22, 2015 at 5:34 PM, John Yost <[email protected] > > <mailto:[email protected]>> wrote: > > > > This is an excellent question, need this info for myself as well. > > > > --John > > > > On Mon, Jun 22, 2015 at 7:40 AM, Hemanth Yamijala > > <[email protected] <mailto:[email protected]>> wrote: > > > > Hi, > > > > I was testing the rebalance functionality on Storm 0.9.4. > > > > storm rebalance <name> -w 10 -n 2 > > - Works as expected. It increased the number of workers to 2. > > > > storm rebalance <name> -w 10 -n 2 -e <bolt-name>=20 > > - Works only for increasing the number of workers, but did *not* > > change the number of executors of <bolt-name>. > > > > I came across this question on > > StackOverflow: > http://stackoverflow.com/questions/18716780/storm-v0-8-2-rebalance-command-not-updating-the-number-of-executors-for-a-bolt > > > > which seems to indicate that the number of executors is bounded > > by the number of tasks, and if number of executors = number of > > tasks (which it is in my case), we can't increase this further. > > > > Is this right ? Is there a way we can increase the number of > > executors at run time without restarting the topology (along > > with increasing number of workers) ? > > > > Thanks > > Hemanth > > > > > > > >
