tasks are run serially by the executor.
On Thu, Apr 3, 2014 at 1:42 PM, Huiliang Zhang <[email protected]> wrote: > Thanks. But how the multiple tasks are executed inside a single executor > thread? in sequential order one by one or the executor thread spawns new > threads for each tasks? > > > On Thu, Apr 3, 2014 at 10:34 AM, Nathan Leung <[email protected]> wrote: > >> by default each task is executed by 1 executor, but if the number of >> tasks is greater than the number of executors, then each executor (thread) >> will execute more than one task. Note that when rebalancing a topology, >> you can change the number of executors and the number of workers, but not >> the number of tasks. >> >> >> On Thu, Apr 3, 2014 at 1:31 PM, Huiliang Zhang <[email protected]> wrote: >> >>> >>> http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/ >>> is >>> a very good article about the running of topology. I have another question: >>> >>> Since executor is in fact thread in the worker process, what's task >>> inside an executor thread? We can see that there may be several tasks for a >>> same component inside a single executor thread. How will multiple tasks be >>> executed inside the executor thread? >>> >>> >>> On Wed, Apr 2, 2014 at 9:25 PM, padma priya chitturi < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> This is how you should run nimbus/supervisor: >>>> >>>> /bin$./storm nimbus >>>> /bin$./storm supervisor >>>> >>>> >>>> On Wed, Apr 2, 2014 at 11:42 PM, Leonardo Bohac < >>>> [email protected]> wrote: >>>> >>>>> Hello, I've downloaded the last version of storm at >>>>> http://storm.incubator.apache.org/downloads.html and when I try to do >>>>> the */bin/storm nimbus* command I get the following message: >>>>> >>>>> *The storm client can only be run from within a release. You appear to >>>>> be trying to run the client from a checkout of Storm's source code.* >>>>> >>>>> *You can download a Storm release >>>>> at http://storm-project.net/downloads.html >>>>> <http://storm-project.net/downloads.html>* >>>>> >>>>> >>>>> >>>>> I don't know whats missing... >>>>> >>>>> >>>>> Thanks! >>>>> >>>>> >>>>> 2014-04-02 15:05 GMT-03:00 Nathan Leung <[email protected]>: >>>>> >>>>> No, it creates an extra executor to deal with processing the ack >>>>>> messages that are sent by the bolts after processing tuples. See the >>>>>> following for details on how acking works in storm: >>>>>> https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing. >>>>>> By default storm will create 1 acker per worker you have in your >>>>>> topology. >>>>>> >>>>>> >>>>>> On Wed, Apr 2, 2014 at 2:01 PM, Huiliang Zhang <[email protected]>wrote: >>>>>> >>>>>>> Hi Nathan, >>>>>>> >>>>>>> The last bolt just emits the tuples and no more bolt in the topology >>>>>>> will consume and ack the tuples. Do you mean that storm automatically >>>>>>> creates an extra executor to deal with the tuples? >>>>>>> >>>>>>> Thanks, >>>>>>> Huiliang >>>>>>> >>>>>>> >>>>>>> On Wed, Apr 2, 2014 at 8:31 AM, Nathan Leung <[email protected]>wrote: >>>>>>> >>>>>>>> the extra task/executor is the acker thread. >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Apr 1, 2014 at 9:23 PM, Huiliang Zhang <[email protected]>wrote: >>>>>>>> >>>>>>>>> I just submitted ExclamationTopology for testing. >>>>>>>>> >>>>>>>>> builder.setSpout("word", new TestWordSpout(), 10); >>>>>>>>> >>>>>>>>> builder.setBolt("exclaim1", new ExclamationBolt(), >>>>>>>>> 3).shuffleGrouping("word"); >>>>>>>>> >>>>>>>>> builder.setBolt("exclaim2", new ExclamationBolt(), >>>>>>>>> 2).shuffleGrouping("exclaim1"); >>>>>>>>> >>>>>>>>> I am supposed to see 15 executors. However, I see 16 executors and >>>>>>>>> 16 tasks on topology summary on storm UI. The numbers of executors are >>>>>>>>> correct for the specific spout and bolts and aggregate to 15. Is that >>>>>>>>> a bug >>>>>>>>> in displaying topology summary? >>>>>>>>> >>>>>>>>> My cluster consists of 2 supervisors and each has 4 workers >>>>>>>>> defined. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Apr 1, 2014 at 1:43 PM, Nathan Leung <[email protected]>wrote: >>>>>>>>> >>>>>>>>>> By default supervisor nodes can run up to 4 workers. This is >>>>>>>>>> configurable in storm.yaml (for example see supervisor.slots.ports >>>>>>>>>> here: >>>>>>>>>> https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml). >>>>>>>>>> Memory should be split between the workers. It's a typical Java >>>>>>>>>> heap, so >>>>>>>>>> anything running on that worker process shares the heap. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Apr 1, 2014 at 4:10 PM, David Crossland < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> On said subject, how does memory allocation work I these >>>>>>>>>>> cases? Assuming 1 worker per node would you just dump all the memory >>>>>>>>>>> available into worker.childopts? I guess the memory pool would be >>>>>>>>>>> shared >>>>>>>>>>> between the spawned threads as appropriate to their needs? >>>>>>>>>>> >>>>>>>>>>> I'm assuming the equivalent options for supervisor/nimbus are >>>>>>>>>>> fine left at defaults. Given that the workers/spouts/bolts are the >>>>>>>>>>> working >>>>>>>>>>> parts of the topology these would where I should target available >>>>>>>>>>> memory? >>>>>>>>>>> >>>>>>>>>>> D >>>>>>>>>>> >>>>>>>>>>> *From:* Huiliang Zhang <[email protected]> >>>>>>>>>>> *Sent:* Tuesday, 1 April 2014 19:47 >>>>>>>>>>> *To:* [email protected] >>>>>>>>>>> >>>>>>>>>>> Thanks. It should be good if there exist some example figures >>>>>>>>>>> explaining the relationship between tasks, workers, and threads. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> No, a single worker is dedicated to a single topology no matter >>>>>>>>>>>> how >>>>>>>>>>>> many threads it spawns for different bolts/spouts. >>>>>>>>>>>> A single worker cannot be shared across multiple topologies. >>>>>>>>>>>> >>>>>>>>>>>> On 3/29/14, Nathan Leung <[email protected]> wrote: >>>>>>>>>>>> > From what I have seen, the second topology is run with 1 >>>>>>>>>>>> worker until you >>>>>>>>>>>> > kill the first topology or add more worker slots to your >>>>>>>>>>>> cluster. >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> > >>>>>>>>>>>> >> Thanks. I am still not clear. >>>>>>>>>>>> >> >>>>>>>>>>>> >> Do you mean that in a single worker process, there will be >>>>>>>>>>>> multiple >>>>>>>>>>>> >> threads and each thread will handle part of a topology? If >>>>>>>>>>>> so, what does >>>>>>>>>>>> >> the number of workers mean when submitting topology? >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi < >>>>>>>>>>>> >> [email protected]> wrote: >>>>>>>>>>>> >> >>>>>>>>>>>> >>> Hi, >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> No, its not the case. No matter how many topologies you >>>>>>>>>>>> submit, the >>>>>>>>>>>> >>> workers will be shared among the topologies. >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> Thanks, >>>>>>>>>>>> >>> Padma Ch >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> On Sat, Mar 29, 2014 at 5:11 AM, Huiliang Zhang < >>>>>>>>>>>> [email protected]> >>>>>>>>>>>> >>> wrote: >>>>>>>>>>>> >>> >>>>>>>>>>>> >>>> Hi, >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> I have a simple question about storm. >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> My cluster has just 1 supervisor and 4 ports are defined >>>>>>>>>>>> to run 4 >>>>>>>>>>>> >>>> workers. I first submit a topology which needs 3 workers. >>>>>>>>>>>> Then I submit >>>>>>>>>>>> >>>> another topology which needs 2 workers. Does this mean >>>>>>>>>>>> that the 2nd >>>>>>>>>>>> >>>> topology will never be run? >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> Thanks, >>>>>>>>>>>> >>>> Huiliang >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> >> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
