[
https://issues.apache.org/jira/browse/FLINK-39523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Fan updated FLINK-39523:
----------------------------
Description:
Complete the memory-based implementation of checkpointing during recovery.
A new RecoveryCheckpointTrigger is dispatched by the barrier handlers at
checkpoint start: it inserts a RecoveryCheckpointBarrier into every channel
still in recovery, so checkpointStarted's in-recovery branch can persist
exactly the pre-barrier recovered data (required even with all data already in
the channel queues — a barrier arriving on an already-recovered channel fans
checkpointStarted out to sibling channels still in recovery, which must find
the marker or decline).
Scope:
* RecoveryCheckpointTrigger with NO_OP and NOT_READY (declines as
CHECKPOINT_DECLINED_TASK_NOT_READY while filtering runs) plus a
barrier-inserting in-memory implementation (the one-shot in-memory push leaves
no undrained residue, so there is no snapshot to transfer).
* ChannelState dispatches checkpoint start through a single hook (trigger step
+ per-input checkpointStarted); AlternatingCollectingBarriers /
AlternatingWaitingForFirstBarrierUnaligned switch to it.
* StreamTask trigger lifecycle: NOT_READY while filtering → barrier-inserting
trigger during consumption → NO_OP once all gates report state consumed; all
transitions on the mailbox thread.
* Restore execution.checkpointing.during-recovery.enabled to ITCase
randomization and re-enable the dedicated filtering ITCase — the randomized
ITCase fleet now exercises the feature in ~50% of runs.
was:Adds an `addInputData(checkpointId, channelInfo, startSeqNum,
InputStream, dataLength)` overload to `ChannelStateWriter`, along with the
matching `ChannelStateWriteRequest` factory and a
`ChannelStateCheckpointWriter.writeInputStreaming()` path that copies bytes via
`InputStream.transferTo()`. This lets spilled recovery data be checkpointed
without allocating Network or heap buffers. The on-disk write format (`[4B
length][data]`) and the recovery read path are unchanged.
> Support checkpoint coordination during recovery via a recovery-checkpoint
> trigger (memory-based)
> ------------------------------------------------------------------------------------------------
>
> Key: FLINK-39523
> URL: https://issues.apache.org/jira/browse/FLINK-39523
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
> Reporter: Rui Fan
> Assignee: Rui Fan
> Priority: Major
>
> Complete the memory-based implementation of checkpointing during recovery.
> A new RecoveryCheckpointTrigger is dispatched by the barrier handlers at
> checkpoint start: it inserts a RecoveryCheckpointBarrier into every channel
> still in recovery, so checkpointStarted's in-recovery branch can persist
> exactly the pre-barrier recovered data (required even with all data already
> in the channel queues — a barrier arriving on an already-recovered channel
> fans checkpointStarted out to sibling channels still in recovery, which must
> find the marker or decline).
> Scope:
> * RecoveryCheckpointTrigger with NO_OP and NOT_READY (declines as
> CHECKPOINT_DECLINED_TASK_NOT_READY while filtering runs) plus a
> barrier-inserting in-memory implementation (the one-shot in-memory push
> leaves no undrained residue, so there is no snapshot to transfer).
> * ChannelState dispatches checkpoint start through a single hook (trigger
> step + per-input checkpointStarted); AlternatingCollectingBarriers /
> AlternatingWaitingForFirstBarrierUnaligned switch to it.
> * StreamTask trigger lifecycle: NOT_READY while filtering →
> barrier-inserting trigger during consumption → NO_OP once all gates report
> state consumed; all transitions on the mailbox thread.
> * Restore execution.checkpointing.during-recovery.enabled to ITCase
> randomization and re-enable the dedicated filtering ITCase — the randomized
> ITCase fleet now exercises the feature in ~50% of runs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
