Gen Luo created FLINK-39835:
-------------------------------
Summary: AsyncWaitOperator supports soft backpressure to avoid
blocking unaligned checkpoint
Key: FLINK-39835
URL: https://issues.apache.org/jira/browse/FLINK-39835
Project: Flink
Issue Type: Improvement
Components: Runtime / Task
Reporter: Gen Luo
When AsyncWaitOperator processes input, if the async queue is full, it yields
and waits for a callback to complete to free up a slot. During this time,
unaligned checkpoints cannot be processed.
Soft backpressure is used to ensure UC can be processed in time. In the
non-async path, processInput produces data written to the ResultPartition. If
the RP is full, processInput can be blocked by soft backpressure so that mails
can be executed.
The same idea can apply to AsyncWaitOperator: on the async path, processInput
produces requests written to the AsyncWaitOperator's queue — when the queue is
full, processInput should be blocked. Similarly, results from AsyncWaitOperator
produce data written to downstream (e.g., RP) — if downstream is full,
AsyncWaitOperator's emitResult should also be blocked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
