Do you have max spout pending set in the topology? It could be that your spout is throttling data since there are too many tuples in flight simultaneously, which will have the effect of reducing your maximum throughput.
Whether you are reaching framework communications limits depends on your application (how many tuples you are sending), and your network. If you suspect this to be the case, you can turn some shuffleGroupings (if you have any) into localOrShuffleGroupings, which try to keep the data in process if possible, thereby reducing network congestion. On Fri, Dec 19, 2014 at 12:16 PM, Jens-U. Mozdzen <[email protected]> wrote: > > Hi, > > we have a topology with 34 bolts and a single spout, running 101 tasks in > 101 executors. > > When running the topology in a single worker, we reach a certain > throughput "x". > > Since some of the bolts will be rather memory-intensive, we decided to > split the topology across 4 workers, on four physically separated server > machines. > > When running that 4-worker setup, we see only a slight increase of > throughput. None of the servers is under heavy CPU load and all bolts have > a capacity below .5, according to the web ui. > > The spout implementation is able to easily serve 100 times more tuples, so > it's not a problem of "starvation" at that level. > > Do I see it right that the bottleneck is not within the topology > implementation, but in the inter-worker communications setup? Else I'd have > expected to see some bolt to run at 1.0 or higher capacity. > > (We had run the same test on older hardware, where the topology was able > to consume all available CPU - there, spreading across more hardware lead > to a significant throughput increase, and "capacity" was reported as >1 for > some bolts.) > > How would I proceed to identify the actual bottleneck(s)? > > Regards, > Jens > >
