GGraziadei commented on PR #8892: URL: https://github.com/apache/storm/pull/8892#issuecomment-4937394440
Thanks @reiabreu, your intuition was entirely correct. Using a streaming grouping driven by a feedback control loop is fundamental to decoupling the control plane from the data plane. In particular, enabling `JitterAwareStreamingGroup` with a dedicated control queue (CQ) is a game-changer (details are in the attached report). Here are the key takeaways from the performance scenarios: - Under No-Backpressure Conditions (No Jitter Asymmetry) `FileReadWordCountTopo`: Enabling a dedicated control queue introduces an unnecessary overhead. Because the single-threaded implementation must poll the control queue before the data queue, data plane performance is penalized linearly relative to traffic volume in control plane (e.g., metrics or ticks). In this scenario, the control queue provides no actionable data to optimize the data plane. - Under Backpressured Conditions: Enabling the control queue acts as an invariant. The system's performance bottleneck is driven entirely by the backpressure itself, rather than the minor overhead of the thread polling the control queue. [Jitter-Aware Stream Grouping_ Decoupling the Control Queue.pdf](https://github.com/user-attachments/files/29897718/Jitter-Aware.Stream.Grouping_.Decoupling.the.Control.Queue.pdf) [Evaluating the Effect of the Control Queue in a Backpressured Scenario.pdf](https://github.com/user-attachments/files/29897719/Evaluating.the.Effect.of.the.Control.Queue.in.a.Backpressured.Scenario.pdf) [Evaluating the cost of a separate Control Queue.pdf](https://github.com/user-attachments/files/29897720/Evaluating.the.cost.of.a.separate.Control.Queue.pdf) Raw data: [BackPressureTopo.txt](https://github.com/user-attachments/files/29897737/BackPressureTopo.txt) [FileReadWordCountTopo.txt](https://github.com/user-attachments/files/29897738/FileReadWordCountTopo.txt) [JitterAwareGroupingTopology.txt](https://github.com/user-attachments/files/29897740/JitterAwareGroupingTopology.txt) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
