Hi Nathan, Tried increasing the number of workers with no luck. Also increased the parallelism of the throughput measuring bolt and checked the aggregate throughput, but that does not seem to be the bottleneck. Also the capacity value of the throughput measuring bolt in Storm UI is at around ~0.12.
Will try out more configurations and see. Thank you very much for your tips. Any other tweaks I might try out? Thanks, Lasantha On Tue, May 13, 2014 at 6:38 PM, Nathan Leung <[email protected]> wrote: > For 20 spouts and even number of processing bolts 3 seems like an odd > number of workers. Also are you sure you're not bottlenecked by your > throughput measuring bolt? > On May 13, 2014 2:43 AM, "Lasantha Fernando" <[email protected]> > wrote: > >> Hi all, >> >> Is there any guide or hints on how to configure storm to scale better? >> >> I was running some tests with a custom scheduler and found that the >> throughput did not scale as expected. Any pointers on what I am doing wrong? >> >> Parallelism 2 4 8 16 Single Node (Avg) 166099 161539.5 193986 N/A Two >> Node (Avg) 160988 165563 174675.5 177624.5 >> The topology is as follows. >> >> Spout (Generates events continuously) -> Processing Bolt -> Throughput >> Measurement Bolt >> >> Parallelism is varied for the processing bolt. >> >> Parallelism for spout and throughput measuring bolt is kept constant at >> 20 and 1 respectively. >> >> Topology.NUM_WORKERS = 3 >> >> Custom scheduler code is available at [1]. Topology code is available at >> [2]. Any pointers would be much appreciated. >> >> Thanks, >> Lasantha >> >> [1] >> https://github.com/sajithshn/storm-schedulers/blob/master/src/main/java/org/wso2/siddhi/storm/scheduler/RoundRobinStormScheduler.java >> [2] >> https://github.com/lasanthafdo/siddhi-storm/blob/master/src/main/java/org/wso2/siddhi/storm/StockDataTopology.java >> >
