Each executor is a separate JVM thread, so in some cases you may want to conserve threads. Consider a case where you have many tasks, some of which are very light weight. In that case, it may make sense to group the lightweight tasks under one executor/thread. JVM threads come with a certain amount of overhead.
It's largely a tuning situation. In many cases the defaults are fine. But we expose it to developers for the situations where the defaults might not be the best. The fact that flux doesn't support it is a simple oversight -- core functionality was the initial focus. I'll look into adding it to flux, but would also support any pull request that added the capability. (Adding dev@ to include anyone who might be interested.) -Taylor > On May 13, 2017, at 7:24 PM, S G <sg.online.em...@gmail.com> wrote: > > Thanks Anshu. > If you can provide an example, that would be much appreciated. > >> On Sat, May 13, 2017 at 1:24 PM, anshu shukla <anshushuk...@gmail.com> wrote: >> The main reason behind task in an executor is dynamic scaling based on input >> rate/ resource req. Using storm rebalance. >> >>> On Sun, May 14, 2017 at 12:52 AM, S G <sg.online.em...@gmail.com> wrote: >>> >>> Hi, >>> >>> As per this guide: >>> http://storm.apache.org/releases/current/Understanding-the-parallelism-of-a-Storm-topology.html, >>> >>> An executor is a thread containing many tasks. >>> And each task is an instance of a spout or a bolt. >>> >>> I cannot imagine when would someone want to have multiple tasks in an >>> executor. >>> AFAIK, Only use-case possible is when the tasks in an executor have to be >>> synchronous i.e. first task must wait for the second task (in the same >>> executor) to complete and that must wait for the third task (in the same >>> executor) to complete and so on. >>> >>> But that requires the tasks in an executor to be different from each other. >>> Where as the current API does not permit tasks of different type within an >>> executor. >>> >>> So I am "guessing" that multiple tasks in an executor is just a historical >>> artifact with no real use-case. >>> And that is also a reason why Flux has no option to specify num-tasks. >>> >>> $ grep -ri task storm/flux/flux-core/src >>> ===> nothing ! >>> >>> Please confirm if that is a correct understanding. >>> Also, if that is correct, we might be able to squeeze some performance by >>> allowing only a single task per executor. >>> >>> Thanks >>> SG >>> >>> >> >> >> >> -- >> Thanks & Regards, >> Anshu Shukla >
