Hi Dimitris, 1. how many topic partitions you've 2. make sure you are distributing data from kafka producer side into all of these partitions 3. whats your kafakspout parallelism set to 4. whats you topology.max.spout.pending set to 5. if you can , briefly describe what topology is doing. 6. are you seeing anything under failed column in Stom UI. 7. any errors in storm topology logs.
Thanks, Harsha On Sat, Jul 25, 2015, at 05:29 AM, Dimitris Sarlis wrote: > Hi all, > > I'm trying to run a topology in Storm and I am facing some scalability > issues. Specifically, I have a topology where KafkaSpouts read from a > Kafka queue and emit messages to bolts which are connected with each > other through directGrouping. (Each bolt is connected with itself as > well as with each one of the other bolts). Spouts subscribe to bolts > with shuffleGrouping. I observe that when I increase the number of > spouts and bolts proportionally, I don't get the speedup I'm expecting > to. In fact, my topology seems to run slower and for the same amount of > data, it takes more time to complete. For example, when I increase > spouts from 4->8 and bolts from 4->8, it takes longer to process the > same amount of kafka messages. > > Any ideas why this is happening? Thanks in advance. > > Best, > Dimitris Sarlis
