[
https://issues.apache.org/jira/browse/FLINK-34704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835796#comment-17835796
]
Piotr Nowojski commented on FLINK-34704:
----------------------------------------
Adding new {{yield}} method wouldn't help, as we can not perform a checkpoint
while operator is yielding, since that can lead to inconsistent state for
upstream operators. For example when upstream operator does this:
{code}
OperatorXYZ::processElement(e, output) {
stateA = e.foo;
output.collect(e);
stateB = e.bar;
}
{code}
Chained operator inside `output.collect(e)` call, can not allow for checkpoint
to be executed, since this would checkpoint the {{OperatorXYZ}} in an
inconsistent state.
I've discovered this issue independently (but for other operators) and
described the problem in FLINK-35051. I think a better solution would be to
make {{StreamTask}} prioritise:
* processing priority messages from network stack over processing mails
* execute out of order "urgent" mails, like time outing aligned checkpoint
barriers to unaligned ones or triggerCheckpoint RPCs for source tasks
I would propose to close this ticket for just {{AsyncWaitOperator}} in favour
of FLINK-35051.
> Process checkpoint barrier in AsyncWaitOperator when the element queue is full
> ------------------------------------------------------------------------------
>
> Key: FLINK-34704
> URL: https://issues.apache.org/jira/browse/FLINK-34704
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: Zakelly Lan
> Priority: Minor
>
> As discussed in
> https://lists.apache.org/thread/4f7ywn29kdv4302j2rq3fkxc6pc8myr2 . Maybe it
> is better to provide such a new `yield` that can process mail with low
> priority in the mailbox executor. More discussion needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
