[
https://issues.apache.org/jira/browse/FLINK-34704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836106#comment-17836106
]
Gyula Fora edited comment on FLINK-34704 at 4/11/24 10:41 AM:
--------------------------------------------------------------
I agree with [~pnowojski] here, the currently blocked element would be lost in
the checkpoint. But [~Zakelly] also has a valid point.
I have played around with this and there is a simple optimisation to be made
for the async operator though under certain circumstances.
If the AWOP is the head of the operator chain (no upstream), we could actually
checkpoint during yielding but we would also need to checkpoint the current
processed element as part of the buffer (temporarily increase the size of the
buffer by 1).
This is still related to the other ticket in the sense that we need to get the
checkpoint trigger during yield but it needs a custom logic for the AWOP to
allow checkpointing while being blocked on the full buffer
was (Author: gyfora):
I agree with [~pnowojski] here, the currently blocked element would be lost in
the checkpoint.
I have played around with this and there is a simple optimisation to be made
for the async operator though under certain circumstances.
If the AWOP is the head of the operator chain (no upstream), we could actually
checkpoint during yielding but we would also need to checkpoint the current
processed element as part of the buffer (temporarily increase the size of the
buffer by 1).
This is still related to the other ticket in the sense that we need to get the
checkpoint trigger during yield but it needs a custom logic for the AWOP to
allow checkpointing while being blocked on the full buffer
> Process checkpoint barrier in AsyncWaitOperator when the element queue is full
> ------------------------------------------------------------------------------
>
> Key: FLINK-34704
> URL: https://issues.apache.org/jira/browse/FLINK-34704
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: Zakelly Lan
> Priority: Minor
>
> As discussed in
> https://lists.apache.org/thread/4f7ywn29kdv4302j2rq3fkxc6pc8myr2 . Maybe it
> is better to provide such a new `yield` that can process mail with low
> priority in the mailbox executor. More discussion needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
