Rui Fan created FLINK-40526:
-------------------------------
Summary: Checkpointing-during-recovery crashes a recovering
RemoteInputChannel when announcing a stashed timeoutable-aligned barrier
Key: FLINK-40526
URL: https://issues.apache.org/jira/browse/FLINK-40526
Project: Flink
Issue Type: Sub-task
Components: Runtime / Checkpointing
Reporter: Rui Fan
During recovery a RemoteInputChannel receives events from the upstream
creditless. In onBuffer, while inRecovery a non-priority event is stashed into
recoveryEventStash and is not added to receivedBuffers, yet announce() is still
called for it because the requiresAnnouncement check sits outside the
inRecovery branch [1]. announce() runs checkAnnouncedOnlyOnce, which scans only
receivedBuffers; because the barrier was stashed it is found 0 times, so
checkState(count == 1) throws IllegalStateException on the Netty thread and
fails the task [2].
The only DataType with requiresAnnouncement is
TIMEOUTABLE_ALIGNED_CHECKPOINT_BARRIER [3], produced when unaligned checkpoints
are enabled with aligned-checkpoint-timeout > 0. It is an event, so it is
delivered to the recovering channel with zero credit. With CDR enabled, a
checkpoint triggered during recovery makes the running upstream inject exactly
this barrier into the still-recovering downstream channel, so every such
checkpoint in the recovery window reproduces the crash, giving a
recover-crash-recover failover loop.
Trigger (all required): CDR enabled + unaligned checkpoints enabled +
aligned-checkpoint-timeout > 0 + a task still recovering channel state
(inRecovery) + the barrier arriving before recovery finishes. When
aligned-checkpoint-timeout is 0 (immediate unaligned) the barrier is a priority
event and never takes the stash-then-announce path, so the crash cannot occur.
[1]
[https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java#L836-L845]
[2]
[https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java#L906-L918]
[3]
[https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/Buffer.java#L305]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
