by default each task is executed by 1 executor, but if the number of tasks is greater than the number of executors, then each executor (thread) will execute more than one task. Note that when rebalancing a topology, you can change the number of executors and the number of workers, but not the number of tasks.
On Thu, Apr 3, 2014 at 1:31 PM, Huiliang Zhang <[email protected]> wrote: > > http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/ > is > a very good article about the running of topology. I have another question: > > Since executor is in fact thread in the worker process, what's task inside > an executor thread? We can see that there may be several tasks for a same > component inside a single executor thread. How will multiple tasks be > executed inside the executor thread? > > > On Wed, Apr 2, 2014 at 9:25 PM, padma priya chitturi < > [email protected]> wrote: > >> Hi, >> >> This is how you should run nimbus/supervisor: >> >> /bin$./storm nimbus >> /bin$./storm supervisor >> >> >> On Wed, Apr 2, 2014 at 11:42 PM, Leonardo Bohac <[email protected] >> > wrote: >> >>> Hello, I've downloaded the last version of storm at >>> http://storm.incubator.apache.org/downloads.html and when I try to do >>> the */bin/storm nimbus* command I get the following message: >>> >>> *The storm client can only be run from within a release. You appear to >>> be trying to run the client from a checkout of Storm's source code.* >>> >>> *You can download a Storm release >>> at http://storm-project.net/downloads.html >>> <http://storm-project.net/downloads.html>* >>> >>> >>> >>> I don't know whats missing... >>> >>> >>> Thanks! >>> >>> >>> 2014-04-02 15:05 GMT-03:00 Nathan Leung <[email protected]>: >>> >>> No, it creates an extra executor to deal with processing the ack >>>> messages that are sent by the bolts after processing tuples. See the >>>> following for details on how acking works in storm: >>>> https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing. >>>> By default storm will create 1 acker per worker you have in your topology. >>>> >>>> >>>> On Wed, Apr 2, 2014 at 2:01 PM, Huiliang Zhang <[email protected]>wrote: >>>> >>>>> Hi Nathan, >>>>> >>>>> The last bolt just emits the tuples and no more bolt in the topology >>>>> will consume and ack the tuples. Do you mean that storm automatically >>>>> creates an extra executor to deal with the tuples? >>>>> >>>>> Thanks, >>>>> Huiliang >>>>> >>>>> >>>>> On Wed, Apr 2, 2014 at 8:31 AM, Nathan Leung <[email protected]>wrote: >>>>> >>>>>> the extra task/executor is the acker thread. >>>>>> >>>>>> >>>>>> On Tue, Apr 1, 2014 at 9:23 PM, Huiliang Zhang <[email protected]>wrote: >>>>>> >>>>>>> I just submitted ExclamationTopology for testing. >>>>>>> >>>>>>> builder.setSpout("word", new TestWordSpout(), 10); >>>>>>> >>>>>>> builder.setBolt("exclaim1", new ExclamationBolt(), >>>>>>> 3).shuffleGrouping("word"); >>>>>>> >>>>>>> builder.setBolt("exclaim2", new ExclamationBolt(), >>>>>>> 2).shuffleGrouping("exclaim1"); >>>>>>> >>>>>>> I am supposed to see 15 executors. However, I see 16 executors and >>>>>>> 16 tasks on topology summary on storm UI. The numbers of executors are >>>>>>> correct for the specific spout and bolts and aggregate to 15. Is that a >>>>>>> bug >>>>>>> in displaying topology summary? >>>>>>> >>>>>>> My cluster consists of 2 supervisors and each has 4 workers defined. >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> >>>>>>> On Tue, Apr 1, 2014 at 1:43 PM, Nathan Leung <[email protected]>wrote: >>>>>>> >>>>>>>> By default supervisor nodes can run up to 4 workers. This is >>>>>>>> configurable in storm.yaml (for example see supervisor.slots.ports >>>>>>>> here: >>>>>>>> https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml). >>>>>>>> Memory should be split between the workers. It's a typical Java >>>>>>>> heap, so >>>>>>>> anything running on that worker process shares the heap. >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Apr 1, 2014 at 4:10 PM, David Crossland < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> On said subject, how does memory allocation work I these cases? >>>>>>>>> Assuming 1 worker per node would you just dump all the memory >>>>>>>>> available >>>>>>>>> into worker.childopts? I guess the memory pool would be shared >>>>>>>>> between the >>>>>>>>> spawned threads as appropriate to their needs? >>>>>>>>> >>>>>>>>> I'm assuming the equivalent options for supervisor/nimbus are >>>>>>>>> fine left at defaults. Given that the workers/spouts/bolts are the >>>>>>>>> working >>>>>>>>> parts of the topology these would where I should target available >>>>>>>>> memory? >>>>>>>>> >>>>>>>>> D >>>>>>>>> >>>>>>>>> *From:* Huiliang Zhang <[email protected]> >>>>>>>>> *Sent:* Tuesday, 1 April 2014 19:47 >>>>>>>>> *To:* [email protected] >>>>>>>>> >>>>>>>>> Thanks. It should be good if there exist some example figures >>>>>>>>> explaining the relationship between tasks, workers, and threads. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> No, a single worker is dedicated to a single topology no matter >>>>>>>>>> how >>>>>>>>>> many threads it spawns for different bolts/spouts. >>>>>>>>>> A single worker cannot be shared across multiple topologies. >>>>>>>>>> >>>>>>>>>> On 3/29/14, Nathan Leung <[email protected]> wrote: >>>>>>>>>> > From what I have seen, the second topology is run with 1 worker >>>>>>>>>> until you >>>>>>>>>> > kill the first topology or add more worker slots to your >>>>>>>>>> cluster. >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> > >>>>>>>>>> >> Thanks. I am still not clear. >>>>>>>>>> >> >>>>>>>>>> >> Do you mean that in a single worker process, there will be >>>>>>>>>> multiple >>>>>>>>>> >> threads and each thread will handle part of a topology? If so, >>>>>>>>>> what does >>>>>>>>>> >> the number of workers mean when submitting topology? >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi < >>>>>>>>>> >> [email protected]> wrote: >>>>>>>>>> >> >>>>>>>>>> >>> Hi, >>>>>>>>>> >>> >>>>>>>>>> >>> No, its not the case. No matter how many topologies you >>>>>>>>>> submit, the >>>>>>>>>> >>> workers will be shared among the topologies. >>>>>>>>>> >>> >>>>>>>>>> >>> Thanks, >>>>>>>>>> >>> Padma Ch >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> On Sat, Mar 29, 2014 at 5:11 AM, Huiliang Zhang < >>>>>>>>>> [email protected]> >>>>>>>>>> >>> wrote: >>>>>>>>>> >>> >>>>>>>>>> >>>> Hi, >>>>>>>>>> >>>> >>>>>>>>>> >>>> I have a simple question about storm. >>>>>>>>>> >>>> >>>>>>>>>> >>>> My cluster has just 1 supervisor and 4 ports are defined to >>>>>>>>>> run 4 >>>>>>>>>> >>>> workers. I first submit a topology which needs 3 workers. >>>>>>>>>> Then I submit >>>>>>>>>> >>>> another topology which needs 2 workers. Does this mean that >>>>>>>>>> the 2nd >>>>>>>>>> >>>> topology will never be run? >>>>>>>>>> >>>> >>>>>>>>>> >>>> Thanks, >>>>>>>>>> >>>> Huiliang >>>>>>>>>> >>>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >> >>>>>>>>>> > >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
