Roman Khachatryan created FLINK-41547:
-----------------------------------------

             Summary: Operator events can miss checkpoints in tasks with 
chained sources, because network barriers are not ordered with queued events
                 Key: FLINK-41547
                 URL: https://issues.apache.org/jira/browse/FLINK-41547
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Checkpointing
            Reporter: Roman Khachatryan


_(similar to FLINK-41442 but older and narrower; inferred from code analysis)_

 

*Problem*

When a task has a chained FLIP-27 source and network inputs 
(MultipleInputStreamTask), an operator event such as AddSplitEvent can be 
enqueued in the mailbox before a checkpoint barrier, and the chained source can 
still be snapshotted before the event is applied. The enumerator's checkpoint 
already records the split as assigned, while the reader's state doesn't contain 
it. On restore, the split is never read again, so data is lost.

The coordinator assumes that an event is part of the next checkpoint once the 
event is enqueued in the task's mailbox. OperatorCoordinatorHolder waits for 
all pending events to be delivered before it completes its own checkpoint. For 
a FLIP-27 source task, this holds because the checkpoint trigger is also a 
mailbox mail and is ordered after the event, as long as both run in order (see 
FLINK-41442). In a task with chained sources, though, the snapshot can be 
triggered by a barrier from a network input, and that barrier doesn't go 
through the mailbox in order with the event.

 

*Scenario 1: stale priority-event mail*

CheckpointedInputGate.waitForPriorityEvents enqueues a mail that drains 
whatever priority events are available when the mail runs. The mail can go 
stale: the input path may poll the priority event that triggered it before the 
mail runs, and processPriorityEvents acknowledges this.

1. Unaligned barrier N arrives and gate mail M is enqueued. The task's input 
processing polls barrier N inline, so M is stale.
2. The task thread is busy for a long time, for example on a slow record or a 
long synchronous snapshot. Meanwhile checkpoint N is aborted. The coordinator 
checkpoints N+1, and AddSplitEvent E is enqueued after M.
3. Barrier N+1 arrives. M runs before E, processes barrier N+1, and snapshots 
the chained source without E.

 

*Scenario 2: barrier polled by the input processor (suspected, not verified)*

MailboxProcessor alternates between processing a batch of mails and running one 
default action. If E is enqueued after createBatch(), and the default action 
then polls the unaligned barrier N+1 inline from the network gate, the snapshot 
runs before E. With unaligned checkpoints, the first barrier snapshots the 
whole chain, including the chained sources. The window is one default-action 
call, but a record that blocks, for example on output buffers under 
backpressure, can widen it.

 

_*Notes*_
- Neither scenario was caused by FLINK-35051. Both are possible with the gate 
mail as a normal mail.
- Aligned checkpoints are not affected. With chained sources, alignment also 
waits for the source trigger, which is a mailbox mail ordered after the event.
- Possible directions:
  - Apply pending operator events before the chained sources are snapshotted in 
the barrier handler, for example by draining operator-event mails before the 
snapshot.
  - Or, before snapshotting the chained sources, apply the events that were 
delivered before the coordinator's checkpoint.

 

*Related:* FLINK-41442, FLINK-35051, and the discussion in 
https://github.com/apache/flink/pull/29954#discussion_r4579142169.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to