GGraziadei commented on issue #8583: URL: https://github.com/apache/storm/issues/8583#issuecomment-4477545395
Hi @rzo1 and @reiabreu, Thank you both for the valuable insights. Reorder buffer You are absolutely right. Breaking the implicit FIFO behaviour is a major correctness risk. Even if co-location is preserved, altering the sequence of tuples within the same stream introduces race conditions that users shouldn't have to reason about. Given this, I am dropping the consumer-side reorder buffer approach. Shifting to the emission-side heuristics I really like @rzo1's suggestion to use the jitter signal at emit-time. Utilizing downstream jitter status to influence the producer's routing decisions is a much cleaner architecture. It naturally avoids head-of-line blocking and prevents local queues from becoming saturated. To achieve this, I am currently evaluating a strategy centred around implementing a jitter-aware [CustomStreamGrouping](https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/grouping/CustomStreamGrouping.java). Feedback stream Regarding the control signal, I agree with the constraints outlined. The feedback contains only EWMA metrics, thus it has the advantage of being async. I completely agree to decouple from the ack logic. I will ensure the control tuple uses a minimal primitive shape (srcTaskId, processJitter, ...) rather than the heavy v2 metrics struct. As the next step, I will reduce the scope of the draft PR #8593 to successfully submit and persist the feedback signal updates as discussed in the `Feedback stream` section and according to your previous comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
