http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/ is a very good article about the running of topology. I have another question:
Since executor is in fact thread in the worker process, what's task inside an executor thread? We can see that there may be several tasks for a same component inside a single executor thread. How will multiple tasks be executed inside the executor thread? On Wed, Apr 2, 2014 at 9:25 PM, padma priya chitturi <[email protected] > wrote: > Hi, > > This is how you should run nimbus/supervisor: > > /bin$./storm nimbus > /bin$./storm supervisor > > > On Wed, Apr 2, 2014 at 11:42 PM, Leonardo Bohac > <[email protected]>wrote: > >> Hello, I've downloaded the last version of storm at >> http://storm.incubator.apache.org/downloads.html and when I try to do >> the */bin/storm nimbus* command I get the following message: >> >> *The storm client can only be run from within a release. You appear to be >> trying to run the client from a checkout of Storm's source code.* >> >> *You can download a Storm release >> at http://storm-project.net/downloads.html >> <http://storm-project.net/downloads.html>* >> >> >> >> I don't know whats missing... >> >> >> Thanks! >> >> >> 2014-04-02 15:05 GMT-03:00 Nathan Leung <[email protected]>: >> >> No, it creates an extra executor to deal with processing the ack messages >>> that are sent by the bolts after processing tuples. See the following for >>> details on how acking works in storm: >>> https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing. >>> By default storm will create 1 acker per worker you have in your topology. >>> >>> >>> On Wed, Apr 2, 2014 at 2:01 PM, Huiliang Zhang <[email protected]> wrote: >>> >>>> Hi Nathan, >>>> >>>> The last bolt just emits the tuples and no more bolt in the topology >>>> will consume and ack the tuples. Do you mean that storm automatically >>>> creates an extra executor to deal with the tuples? >>>> >>>> Thanks, >>>> Huiliang >>>> >>>> >>>> On Wed, Apr 2, 2014 at 8:31 AM, Nathan Leung <[email protected]> wrote: >>>> >>>>> the extra task/executor is the acker thread. >>>>> >>>>> >>>>> On Tue, Apr 1, 2014 at 9:23 PM, Huiliang Zhang <[email protected]>wrote: >>>>> >>>>>> I just submitted ExclamationTopology for testing. >>>>>> >>>>>> builder.setSpout("word", new TestWordSpout(), 10); >>>>>> >>>>>> builder.setBolt("exclaim1", new ExclamationBolt(), >>>>>> 3).shuffleGrouping("word"); >>>>>> >>>>>> builder.setBolt("exclaim2", new ExclamationBolt(), >>>>>> 2).shuffleGrouping("exclaim1"); >>>>>> >>>>>> I am supposed to see 15 executors. However, I see 16 executors and 16 >>>>>> tasks on topology summary on storm UI. The numbers of executors are >>>>>> correct >>>>>> for the specific spout and bolts and aggregate to 15. Is that a bug in >>>>>> displaying topology summary? >>>>>> >>>>>> My cluster consists of 2 supervisors and each has 4 workers defined. >>>>>> >>>>>> Thanks. >>>>>> >>>>>> >>>>>> On Tue, Apr 1, 2014 at 1:43 PM, Nathan Leung <[email protected]>wrote: >>>>>> >>>>>>> By default supervisor nodes can run up to 4 workers. This is >>>>>>> configurable in storm.yaml (for example see supervisor.slots.ports here: >>>>>>> https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml). >>>>>>> Memory should be split between the workers. It's a typical Java heap, >>>>>>> so >>>>>>> anything running on that worker process shares the heap. >>>>>>> >>>>>>> >>>>>>> On Tue, Apr 1, 2014 at 4:10 PM, David Crossland < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> On said subject, how does memory allocation work I these cases? >>>>>>>> Assuming 1 worker per node would you just dump all the memory available >>>>>>>> into worker.childopts? I guess the memory pool would be shared between >>>>>>>> the >>>>>>>> spawned threads as appropriate to their needs? >>>>>>>> >>>>>>>> I'm assuming the equivalent options for supervisor/nimbus are >>>>>>>> fine left at defaults. Given that the workers/spouts/bolts are the >>>>>>>> working >>>>>>>> parts of the topology these would where I should target available >>>>>>>> memory? >>>>>>>> >>>>>>>> D >>>>>>>> >>>>>>>> *From:* Huiliang Zhang <[email protected]> >>>>>>>> *Sent:* Tuesday, 1 April 2014 19:47 >>>>>>>> *To:* [email protected] >>>>>>>> >>>>>>>> Thanks. It should be good if there exist some example figures >>>>>>>> explaining the relationship between tasks, workers, and threads. >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> No, a single worker is dedicated to a single topology no matter how >>>>>>>>> many threads it spawns for different bolts/spouts. >>>>>>>>> A single worker cannot be shared across multiple topologies. >>>>>>>>> >>>>>>>>> On 3/29/14, Nathan Leung <[email protected]> wrote: >>>>>>>>> > From what I have seen, the second topology is run with 1 worker >>>>>>>>> until you >>>>>>>>> > kill the first topology or add more worker slots to your cluster. >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang < >>>>>>>>> [email protected]> wrote: >>>>>>>>> > >>>>>>>>> >> Thanks. I am still not clear. >>>>>>>>> >> >>>>>>>>> >> Do you mean that in a single worker process, there will be >>>>>>>>> multiple >>>>>>>>> >> threads and each thread will handle part of a topology? If so, >>>>>>>>> what does >>>>>>>>> >> the number of workers mean when submitting topology? >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi < >>>>>>>>> >> [email protected]> wrote: >>>>>>>>> >> >>>>>>>>> >>> Hi, >>>>>>>>> >>> >>>>>>>>> >>> No, its not the case. No matter how many topologies you >>>>>>>>> submit, the >>>>>>>>> >>> workers will be shared among the topologies. >>>>>>>>> >>> >>>>>>>>> >>> Thanks, >>>>>>>>> >>> Padma Ch >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> On Sat, Mar 29, 2014 at 5:11 AM, Huiliang Zhang < >>>>>>>>> [email protected]> >>>>>>>>> >>> wrote: >>>>>>>>> >>> >>>>>>>>> >>>> Hi, >>>>>>>>> >>>> >>>>>>>>> >>>> I have a simple question about storm. >>>>>>>>> >>>> >>>>>>>>> >>>> My cluster has just 1 supervisor and 4 ports are defined to >>>>>>>>> run 4 >>>>>>>>> >>>> workers. I first submit a topology which needs 3 workers. >>>>>>>>> Then I submit >>>>>>>>> >>>> another topology which needs 2 workers. Does this mean that >>>>>>>>> the 2nd >>>>>>>>> >>>> topology will never be run? >>>>>>>>> >>>> >>>>>>>>> >>>> Thanks, >>>>>>>>> >>>> Huiliang >>>>>>>>> >>>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >> >>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
