srdo commented on PR #8787: URL: https://github.com/apache/storm/pull/8787#issuecomment-4698008232
Hi. Not to involve myself too much in the current design discussion, but I just wanted to add a bit of context that is missing from the original ticket, so you don't draw the wrong conclusions from the discussion we had there :) As a reminder, here is the original design: > The idea is to track the anchor ids of each non-system message that enters an executor in/out queue. For the inbound queue, the anchor is no longer in progress when the associated tuple is acked or failed. For the outbound queue (pendingEmits in the Executor), the anchor is no longer in progress when the associated tuple gets flushed from pendingEmits. > Occasionally a thread will check the set of in-progress anchors for the worker and send reset messages for all of them to the relevant ackers. In order to avoid sending too many messages, this thread snapshots the anchor set when it runs, and only sends reset messages for anchors that have been in progress sufficiently long in that worker. > Since there may be more than one tuple per anchor, anchors are tracked as a count in a multiset, rather than just presence in a set. The concern at the time was that adding this tracking of anchor ids would be too expensive. The reason we needed it was that the executor in/out queues did not support looking at the queue contents from the resetter thread, so we needed separate tracking of the anchors, because the resetter thread would be unable to look at the contents of those queues directly. The reason it was abandoned was partially that I lost interest, and partially that there was a side discussion going on regarding redesigning the acking infrastructure entirely in https://issues.apache.org/jira/browse/STORM-3314, which would have required an alternative approach. That redesign was abandoned though. Support for looking at the executor queues was added to JCTools later, in https://github.com/JCTools/JCTools/pull/229. Ignore the title, the change is actually adding iterator support, not an `unorderedSnapshot` method. With those changes, it's possible the original design could be tractable now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
