GGraziadei commented on PR #8593: URL: https://github.com/apache/storm/pull/8593#issuecomment-4701392125
Hi @rzo1, Thanks for the detailed review! I really appreciated the feedback and learned a lot from it. You were entirely right about the initial benchmark being skewed (I am sorry for my previous error). It was heavily impacted by a high failure rate due to timeouts (dropping to ~70 tuples/s). Please discard those previous results. I have since applied your suggested fixes to the benchmark setup and properly tuned the max.spout.pending parameter. Key Updates in this Iteration: - Fallback mechanism: The JitterAwareStreamGrouping now implements P2C and properly falls back to LoadAwareStreamGrouping if stats are missing or insufficient to select a candidate task. - New baseline: I agree that beating round robin was too low a bar! I have re-run the benchmarks using LoadAwareStreamGrouping as the baseline. - Tested scenarios: I evaluated the implementation under varying conditions, including cluster saturation (by tweaking max.spout.pending) as well as simulated network latency and jitter. - Overhead analysis: I evaluated the latency and throughput overhead introduced by the feedback tuples using `storm-perf/FileReadWordCountTopo`. Any potential degradation was too small to be accurately isolated, as the variance falls entirely within the baseline noise floor, making the overhead practically negligible. I have attached the updated reports and raw data below. Looking forward to your thoughts! Thanks again! [jitterAwareVSLoadAware.txt](https://github.com/user-attachments/files/28914321/jitterAwareVSLoadAware.txt) [Jitter-Aware vs. Load-Aware Stream Grouping report.pdf](https://github.com/user-attachments/files/28925617/Jitter-Aware.vs.Load-Aware.Stream.Grouping.report.pdf) [feedbackTupleDegradation.txt](https://github.com/user-attachments/files/28925430/feedbackTupleDegradation.txt) [Feedback tuple effects on latency and troughput.pdf](https://github.com/user-attachments/files/28925431/Feedback.tuple.effects.on.latency.and.troughput.pdf) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
