GGraziadei commented on PR #8796: URL: https://github.com/apache/storm/pull/8796#issuecomment-4760284319
## Performance analysis The benchmarks were run against the `FileReadWordCountTopo` topology, a standard word-count workload from Storm's performance suite, exercising the inter-executor transfer path that JCQueue's producer batching governs. The topology was deployed on a dockerized dev-cluster, providing a reproducible, self-contained Storm environment so that each configuration ran under identical resource and parallelism conditions, with the same fixed topology shape (2 workers, 7 tasks, 7 executors, single spout executor) across all runs so that the producer batch-sizing strategy was the only variable. The results consistently favored dynamic batching: across batch ceilings of 10, 100, and 1000 it matched or beat every static configuration on all three metrics, delivering throughput gains of up to ~9% and average complete-latency reductions of 8–12%, with the largest benefit at small ceilings where static batching faces the sharpest latency-versus-throughput trade-off and no measured downside at a ny setting. An extended 600-second run at ceiling 1000 further showed a brief learning phase after which the AIMD controller converges on a stable optimum, with latency settling near 376 ms within a ~2 ms standard deviation, confirming that the policy discovers a good batch size online and then locks onto it rather than relying on a manually tuned fixed value. Attached raw data and report. [jcqueue-dynamic-batch-size.txt](https://github.com/user-attachments/files/29168573/jcqueue-dynamic-batch-size.txt) [Dynamic Producer Batch Sizing in Apache Storm.pdf](https://github.com/user-attachments/files/29168610/Dynamic.Producer.Batch.Sizing.in.Apache.Storm.pdf) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
