Assuming you are truly cpu bound and not waiting on io, and 100ms / tuple, each core can do 10 tuples / s. Each node can do 160, and 19 nodes can do 3040 / s. So then you have to optimize your software or add more nodes; it's not a storm issue.
If you are doing io and not just purely cpu bound you can add more threads to hide latency and would be able to get higher throughout. On Jan 30, 2015 4:28 AM, "wlqpku" <[email protected]> wrote: > i met the same issue > > Sent from X1 7.0 > > Martin Illecker <[email protected]>编写： > > Hello, > > I'm observing a huge performance problem with my topology. > The topology consists of 5 bolts and two of them are really slow about 60 > ms and 30 ms of execute latency. (please see attached UI screenshot) > > My topology configuration consists of one worker per node and 19 16-core > nodes, which is a total of 19 workers. > Every worker runs 3 executors, one for each of the three fast bolts, 5 > executors for the 60 ms bolt, 3 executors for the 30ms bolt and 3 hidden > threads (acker, worker buffer receive, worker buffer transfer). > This leads to a total number of 14 threads per worker, which is the upper > limit for a 16-core nodes. In my case the CPU is the limiting factor. > > With this configuration I could only measure 2000 to 3000 tuples per > second at the end of this pipeline. I have tried multiple MaxSpoutPending > settings but 2000 to 3000 tuples per second seem to be the maximum. > I think, this might be the upper limit for a total execution latency of > 100ms? > > How can I achieve x0,000 tuples for my topology? Or is this impossible > with these two slow bolts? > Is there anything I could try because horizontal scaling will not solve my > problem. > > Thanks! > > Best regards > Martin > > > > >
