how many ackers? make sure this is set to number of workers. what do you have for max spout pending? we configured to have 5-10 seconds of messages in pending state. (divide 5-10 seconds by avg tuple processing time)
Bert On Wed, May 14, 2014 at 4:35 PM, Lasantha Fernando <[email protected]>wrote: > Hi Nathan, > > Tried increasing the number of workers with no luck. Also increased the > parallelism of the throughput measuring bolt and checked the aggregate > throughput, but that does not seem to be the bottleneck. Also the capacity > value of the throughput measuring bolt in Storm UI is at around ~0.12. > > Will try out more configurations and see. Thank you very much for your > tips. > > Any other tweaks I might try out? > > Thanks, > Lasantha > > > On Tue, May 13, 2014 at 6:38 PM, Nathan Leung <[email protected]> wrote: > >> For 20 spouts and even number of processing bolts 3 seems like an odd >> number of workers. Also are you sure you're not bottlenecked by your >> throughput measuring bolt? >> On May 13, 2014 2:43 AM, "Lasantha Fernando" <[email protected]> >> wrote: >> >>> Hi all, >>> >>> Is there any guide or hints on how to configure storm to scale better? >>> >>> I was running some tests with a custom scheduler and found that the >>> throughput did not scale as expected. Any pointers on what I am doing wrong? >>> >>> Parallelism 2 4 8 16 Single Node (Avg) 166099 161539.5 193986 N/A Two >>> Node (Avg) 160988 165563 174675.5 177624.5 >>> The topology is as follows. >>> >>> Spout (Generates events continuously) -> Processing Bolt -> Throughput >>> Measurement Bolt >>> >>> Parallelism is varied for the processing bolt. >>> >>> Parallelism for spout and throughput measuring bolt is kept constant at >>> 20 and 1 respectively. >>> >>> Topology.NUM_WORKERS = 3 >>> >>> Custom scheduler code is available at [1]. Topology code is available at >>> [2]. Any pointers would be much appreciated. >>> >>> Thanks, >>> Lasantha >>> >>> [1] >>> https://github.com/sajithshn/storm-schedulers/blob/master/src/main/java/org/wso2/siddhi/storm/scheduler/RoundRobinStormScheduler.java >>> [2] >>> https://github.com/lasanthafdo/siddhi-storm/blob/master/src/main/java/org/wso2/siddhi/storm/StockDataTopology.java >>> >> >
