GGraziadei commented on issue #8583: URL: https://github.com/apache/storm/issues/8583#issuecomment-4415336757
Hi @rzo1, thank you for your review on #8586. I have prepared this draft PR https://github.com/apache/storm/pull/8593 as a follow-up. Before moving it to a "Ready" state, I’d like to discuss the architectural approach with you. 1. Feedback Loop To complete the control loop, I have implemented a mechanism to propagate a feedback tuple (containing EWMA statistics) from an task to its upstream tasks. This is not an acksignal; its frequency is user-configurable. I have registered a new stream specifically for this feedback. Upon receiving a feedback tuple, an executor updates the perceived jitter status of its downstream tasks. I am aware that introducing these control tuples consumes bandwidth and may slightly reduce overall throughput. Is this approach, using specialized tuples for control signals, acceptable to you, or would you suggest a different method for exposing downstream jitter status to upstream components? 2. Proactive Backpressure via reorder buffer The second phase of this PR involves consuming the downstream jitter status to modify the executor's queue consumption order. The Idea: implement a small reorder buffer (e.g., capacity of 4 tuples). Reorder tuples based on the jitter of the destination task (persisted in the executor stats) to proactively mitigate backpressure. THis work is done by the consumer implementation. What do you think about this approach? 3. Validation Results To validate this proposal, I implemented a discrete event simulator (available in dev-tools). In a test scenario where a task's queue is filled by 4 producers and writes to 6 downstream tasks, I observed the following: Jitter reduced by ~25–30%. Throughput decreased by ~1.8%. Latency increased by ~2.5%. What are your thoughts on these trade-offs and the overall direction? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
