[
https://issues.apache.org/jira/browse/FLINK-40526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131945#comment-18131945
]
Rui Fan commented on FLINK-40526:
---------------------------------
Checkpointing during recovery is temporarily disabled when
aligned-checkpoint-timeout > 0 in FLINK-41193. This ticket tracks the proper
fix.
> Checkpointing-during-recovery crashes a recovering RemoteInputChannel when
> announcing a stashed timeoutable-aligned barrier
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40526
> URL: https://issues.apache.org/jira/browse/FLINK-40526
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
> Reporter: Rui Fan
> Priority: Major
>
> During recovery a RemoteInputChannel receives events from the upstream
> creditless. In onBuffer, while inRecovery a non-priority event is stashed
> into recoveryEventStash and is not added to receivedBuffers, yet announce()
> is still called for it because the requiresAnnouncement check sits outside
> the inRecovery branch [1]. announce() runs checkAnnouncedOnlyOnce, which
> scans only receivedBuffers; because the barrier was stashed it is found 0
> times, so checkState(count == 1) throws IllegalStateException on the Netty
> thread and fails the task [2].
> The only DataType with requiresAnnouncement is
> TIMEOUTABLE_ALIGNED_CHECKPOINT_BARRIER [3], produced when unaligned
> checkpoints are enabled with aligned-checkpoint-timeout > 0. It is an event,
> so it is delivered to the recovering channel with zero credit. With CDR
> enabled, a checkpoint triggered during recovery makes the running upstream
> inject exactly this barrier into the still-recovering downstream channel, so
> every such checkpoint in the recovery window reproduces the crash, giving a
> recover-crash-recover failover loop.
> Trigger (all required): CDR enabled + unaligned checkpoints enabled +
> aligned-checkpoint-timeout > 0 + a task still recovering channel state
> (inRecovery) + the barrier arriving before recovery finishes. When
> aligned-checkpoint-timeout is 0 (immediate unaligned) the barrier is a
> priority event and never takes the stash-then-announce path, so the crash
> cannot occur.
> [1]
> [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java#L836-L845]
>
> [2]
> [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java#L906-L918]
>
> [3]
> [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/Buffer.java#L305]
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
