[
https://issues.apache.org/jira/browse/FLINK-35051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835142#comment-17835142
]
Peter Vary commented on FLINK-35051:
------------------------------------
FLINK-34704 is one of the ways this issue materializes
> Weird priorities when processing unaligned checkpoints
> ------------------------------------------------------
>
> Key: FLINK-35051
> URL: https://issues.apache.org/jira/browse/FLINK-35051
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Network, Runtime /
> Task
> Affects Versions: 1.16.3, 1.17.2, 1.19.0, 1.18.1
> Reporter: Piotr Nowojski
> Priority: Major
>
> While looking through the code I noticed that `StreamTask` is processing
> unaligned checkpoints in strange order/priority. The end result is that
> unaligned checkpoint `Start Delay` / triggering checkpoints in `StreamTask`
> can be unnecessary delayed by other mailbox actions in the system, like for
> example:
> * processing time timers
> * `AsyncWaitOperator` results
> * ...
> Incoming UC barrier is treated as a priority event by the network stack (it
> will be polled from the input before anything else). This is what we want,
> but polling elements from network stack has lower priority then processing
> enqueued mailbox actions.
> Secondly, if AC barrier timeout to UC, that's done via a mailbox action, but
> this mailbox action is also not prioritised in any way, so other mailbox
> actions could be unnecessarily executed first.
> On top of that there is a clash of two separate concepts here:
> # Mailbox priority. yieldToDownstream - so in a sense reverse to what we
> would like to have for triggering checkpoint, but that only kicks in #yield()
> calls, where it's actually correct, that operator in a middle of execution
> can not yield to checkpoint - it should only yield to downstream.
> # Control mails in mailbox executor - cancellation is done via that, it
> bypasses whole mailbox queue.
> # Priority events in the network stack.
> It's unfortunate that 1. vs 3. has a naming clash, as priority name is used
> in both things, and highest network priority event containing UC barrier,
> when executed via mailbox has actually the lowest mailbox priority.
> Control mails mechanism is a kind of priority mails executed out of order,
> but doesn't generalise well for use in checkpointing.
> This whole thing should be re-worked at some point. Ideally what we would
> like have is that:
> * mail to convert AC barriers to UC
> * polling UC barrier from the network input
> * checkpoint trigger via RPC for source tasks
> should be processed first, with an exception of yieldToDownstream, where
> current mailbox priorities should be adhered.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
