Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/765#issuecomment-148544657
@mjsax
What I saw when testing STORM-855 was that the maximum throughput was cut
almost in half from 10,000 sentences per second to 5,500. But your numbers
showed maximum throughput more than doubling from around 7,960,300 tuples sent
in 30 seconds to 16,347,100 in the same time period (no-acking). And 1,832,160
in 30 seconds to 2,323,580 an increase of 25% with acking.
To me this feels like a contradiction. The only thing I can think of is
that the messaging layer is so scary slow that cutting the maximum throughput
of a worker by half has no impact on the overall performance if it can double
the throughput of the messaging layer, by doing more batching.
This is likely the case, as on the high end 16,347,100 / 30 seconds / 24
workers is about 22,000 tuples per second per worker, where as 5,500 sentences
per second results in about 181,500 total tuples per second/worker being
processed.
I'm just looking for feedback from others on this, but it looks like I need
to do a distributed apples to apples comparison as well to see the impact the
messaging layer has.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
