ParalleismHint is definitely the way to scale up consumption of a Kafka topic. Make sure the parallelism doesn't exceed the number of partitions though and try to keep it balanced.
Not sure how your environment is set up but simply deploy more hosts to the cluster and then use the rebalance option in the storm-ui. If you have allocated enough workers up front they will be spread across the new machines. If you already have one worker per machine before adding the new ones I'm not sure how to scale up without changing the number of workers and re-launching the topology. On Wed, Jul 16, 2014 at 6:33 AM, Josh J <[email protected]> wrote: > Hi, > > I have read over the docs here > <http://storm.incubator.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html> > and this stackoverflow answer > <http://stackoverflow.com/questions/14132871/elastic-storm-topology-storm-hadoop-coexisting>. > Though I'm still not clear how to scale by adding additional physical > machines and processes. The recommendations in both articles suggest to > increase the parallelism which increases the number of threads. This would > assume that the machines running the storm topology have enough capacity to > handle the increases in load the additional threads add. > > My question is how do I add additional physical machines and processes to > a running storm topology? > > Thanks, > Josh >
