Github user roshannaik commented on the issue:
https://github.com/apache/storm/pull/2241
@HeartSaVioR the current master clocks little over 3 mill/sec for the
ConstSpoutNullBolt Topo without ACKing. On the latency end, with ACK enabled i
saw numbers in multiple *seconds* with batchSz=1 for master and touching as low
~30 microseconds with the new patch.
More analysis? I thought I might have spent too much time providing
measurements of various kinds, both in the design docs and here. :-)
Actually I have only presented a few pruned down numbers that I felt were
interesting, in which case I was a bit meticulous to record the numbers. Given
the scope of this patch, meticulously recording and cataloging all the perf
tests that I have done (both high level profiler data & low level hardware perf
counters) and performing the many more tests that I could have done, will take
forever. I felt, ROI was -ve to rigorously pursue some of those measurements
any further.
If you feel some other critical measurements are necessary, bring it up.
Batching is a small aspect of this patch. I had done some quick tests to
determine a default that appears to be a modest sweetspot for high throughput
low latency topos, but not enough to provide a rigorous analysis. As batch size
is very dependent on topology components, deployment size and ack-mode, IMO the
search for a more generally optimal batch size is not important right now.
I think we will have more publish worthy analysis once these changes
stabilize and get more testing.
On a related note, about latency and ACK-mode, during my recent
measurements, I noticed an issue in ACK mode, both in current master and
optimized version, perf degrades with time (for certain batch sizes)...
Something I had not noticed on the old version of master on which I had based
my changes for a long time. I will share these observations soon.
This needs to be looked into before we have an official release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
