[ 
https://issues.apache.org/jira/browse/FLINK-41442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Khachatryan updated FLINK-41442:
--------------------------------------
    Fix Version/s: 2.2.2
                   2.3.1
                   2.4.0

> Unaligned checkpoint trigger overtakes queued AddSplitEvent, splits lost on 
> restore
> -----------------------------------------------------------------------------------
>
>                 Key: FLINK-41442
>                 URL: https://issues.apache.org/jira/browse/FLINK-41442
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Checkpointing
>    Affects Versions: 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.3.0, 2.2.1, 2.1.3
>         Environment: Unaligned checkpoints enabled
>            Reporter: Kaivalya Apte
>            Assignee: Roman Khachatryan
>            Priority: Critical
>             Fix For: 2.2.2, 2.3.1, 2.4.0
>
>
> h3. *Problem*
> Since FLINK-35051, \{{StreamTask#triggerCheckpointAsync}} sends an unaligned 
> checkpoint trigger as an *urgent* mail. Urgent mails run before all queued 
> normal mails. Operator events such as \{{AddSplitEvent}} are normal mails 
> (\{{StreamTask#dispatchOperatorEvent}}).
> {\{OperatorCoordinatorHolder#completeCheckpointOnceEventsAreDone}} sends the 
> trigger only after the task acknowledges all earlier events. The task 
> acknowledges an event when the event is queued, not when it is processed. So 
> the trigger can run before a queued \{{AddSplitEvent}}:
> * The coordinator checkpoint records the split as assigned.
> * The reader checkpoint does not contain the split.
> After a restore from that checkpoint, no reader owns the split. The job stays 
> RUNNING with no error and never reads the split again. Later checkpoints keep 
> the same state, so the loss is permanent.
> h3. *Conditions*
> * Unaligned checkpoints, for example 
> \{{execution.checkpointing.unaligned.enabled=true}} with 
> \{{aligned-checkpoint-timeout=0}}.
> * A checkpoint triggers while split assignments are queued on a busy task, 
> usually the first checkpoint after job start.
> * The job restores from exactly that checkpoint (failover or rescale).
> We observed this on several jobs. Logs show "Assigning splits" before 
> "Triggering checkpoint 1", and "Adding split(s) to reader" before "Completed 
> checkpoint 1". After the restore, the lost sources restore 0 splits.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to