Have you read this article on parallelism of storm - http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/
If not, please read it, it's really good. Coming to your problem, the reason this is happening is because you have too few tasks. The number of tasks remains constant during the lifetime of the topology and increasing the parallelism hint will not increase the number of tasks. By default, the number of tasks is equal to the number of executor threads. So initially, you had only one task per component. When you gave the rebalance command, the number of tasks did not increase and therefore your new executors remained idle. To scale a storm topology, you need to plan in advance and use a lot of tasks. Hope this helps On Fri, Jul 18, 2014 at 3:07 PM, Josh J <[email protected]> wrote: > > what is the number of workers set in topology definition. > > topology.workers was set to 1. So I resubmitted the topology with > topology.workers to 5 and parallelismHint 5 for the spout at the end for > the aggregate > > final TridentState wordCounts = topology.newStream("kafka", > kafkaSpout) > .parallelismHint(5) > .shuffle() > // .each(new Fields("str"), new PrintFilter()) > .each(new Fields("str"), new Split(), new Fields("word")) > .groupBy(new Fields("word")) > .persistentAggregate(new MemoryMapState.Factory(), new > Count(), > new Fields("aggregates_words")).parallelismHint(5); > > Thanks, > Josh > > > On Fri, Jul 18, 2014 at 11:11 AM, Susheel Kumar Gadalay < > [email protected]> wrote: > >> what is the number of workers set in topology definition. >> >> On 7/18/14, Josh J <[email protected]> wrote: >> > Hi, >> > >> > I started my topology with a single supervisor. I then added two more >> > supervisor nodes and tried the storm rebalance option in the storm ui. >> > Though the topology doesn't seem to rebalance or utilize the new >> supervisor >> > nods. I see in the storm ui it does recognize and register the >> additional >> > two supervisor nodes. though the topology is not rebalancing to utilize >> the >> > additional capacity. screenshots attached, any suggestions? >> > >> > Thanks, >> > Josh >> > >> > >
