[
https://issues.apache.org/jira/browse/FLINK-41442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan updated FLINK-41442:
--------------------------------------
Fix Version/s: 2.2.2
2.3.1
2.4.0
> Unaligned checkpoint trigger overtakes queued AddSplitEvent, splits lost on
> restore
> -----------------------------------------------------------------------------------
>
> Key: FLINK-41442
> URL: https://issues.apache.org/jira/browse/FLINK-41442
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.3.0, 2.2.1, 2.1.3
> Environment: Unaligned checkpoints enabled
> Reporter: Kaivalya Apte
> Assignee: Roman Khachatryan
> Priority: Critical
> Fix For: 2.2.2, 2.3.1, 2.4.0
>
>
> h3. *Problem*
> Since FLINK-35051, \{{StreamTask#triggerCheckpointAsync}} sends an unaligned
> checkpoint trigger as an *urgent* mail. Urgent mails run before all queued
> normal mails. Operator events such as \{{AddSplitEvent}} are normal mails
> (\{{StreamTask#dispatchOperatorEvent}}).
> {\{OperatorCoordinatorHolder#completeCheckpointOnceEventsAreDone}} sends the
> trigger only after the task acknowledges all earlier events. The task
> acknowledges an event when the event is queued, not when it is processed. So
> the trigger can run before a queued \{{AddSplitEvent}}:
> * The coordinator checkpoint records the split as assigned.
> * The reader checkpoint does not contain the split.
> After a restore from that checkpoint, no reader owns the split. The job stays
> RUNNING with no error and never reads the split again. Later checkpoints keep
> the same state, so the loss is permanent.
> h3. *Conditions*
> * Unaligned checkpoints, for example
> \{{execution.checkpointing.unaligned.enabled=true}} with
> \{{aligned-checkpoint-timeout=0}}.
> * A checkpoint triggers while split assignments are queued on a busy task,
> usually the first checkpoint after job start.
> * The job restores from exactly that checkpoint (failover or rescale).
> We observed this on several jobs. Logs show "Assigning splits" before
> "Triggering checkpoint 1", and "Adding split(s) to reader" before "Completed
> checkpoint 1". After the restore, the lost sources restore 0 splits.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)