You can create your topology with more tasks than executors, then when the rebalance happens you can add executors. However at the moment you cannot add more tasks to a running topology.
On Thu, Jun 9, 2016 at 8:58 AM, Adrien Carreira <[email protected]> wrote: > I've just create a topology like this : > > builder.setBolt("fetcher", new Fetch()) > .shuffleGrouping("spout"); > > builder.setBolt("extract", new Extract()) > .shuffleGrouping("fetcher"); > > builder.setBolt("indexer", new Indexer()) > .shuffleGrouping("extract"); > > > Means that I've three bolt with One Worker and parrallelism_hint of 1. > > Now, Let's say that I've another machine available, or that I've too many > tuple to process and I need another machine. > > > I've executed this command : > > storm rebalance kairos-who -n 2 -e indexer=2 -e fetcher=2 -e extract=2 > > > But what I've is two worker with : > > worker 1 => Spout + extract > > worker 2 => fetcher + indexer > > > What I would love : > > Worker 1 => Spout + fetcher + extract + indexer > > Worker 2 => Same... > > > I hope I'm clear... > > > > > > > > 2016-06-09 14:47 GMT+02:00 Andrew Xor <[email protected]>: > >> Hello, >> >> I am sorry, but I don't know why you cannot emulate those scale up >> factors by using rebalance; after all it spawns the requested amount of >> workers (in topology) and executors (in spouts/bolts) only bounded by the >> topology_max_task_parallelism. Have you read the article in order to >> understand how parallelism works in storm? >> >> Regards. >> >> On Thu, Jun 9, 2016 at 3:34 PM, Adrien Carreira <[email protected]> >> wrote: >> >>> Yes, >>> >>> But the rebalance command doesn't do what I would like. >>> >>> >>> Let's suppose that I've : >>> >>> SPOUT A (1) => BOLT 1 (1) => BOLT2 (1) => BOLT3 (3) >>> >>> (number is the parallelism hint) >>> It means that If I scale to n worker I would like : >>> >>> SPOUT A (1*n) => BOLT 1 (1*n) => BOLT2 (1*n) => BOLT3 (3*n) >>> >>> >>> But, the storm rebalance keeps the parralisme_hint :/ >>> >>> >>> >>> 2016-06-09 14:29 GMT+02:00 Andrew Xor <[email protected]>: >>> >>>> Hello, >>>> >>>> Why not use the rebalance command? It's well documented here >>>> <http://storm.apache.org/releases/current/Understanding-the-parallelism-of-a-Storm-topology.html> >>>> . >>>> >>>> Regards. >>>> >>>> On Thu, Jun 9, 2016 at 3:22 PM, Adrien Carreira <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> After a month building a topology on storm. I've one question about >>>>> parallelism that I can't answer. >>>>> >>>>> I've developed my topology and tested on a cluster with two nodes. >>>>> >>>>> My parallelism_hint are ok, everything are fine. >>>>> >>>>> My question is, if I need to scale the number of worker in the >>>>> topology to have more worker dooing the same thing how can I achieve that >>>>> without kill/restart the topology >>>>> >>>>> Thanks for your reply >>>>> >>>> >>>> >>> >> >
