Roman Khachatryan created FLINK-41547:
-----------------------------------------
Summary: Operator events can miss checkpoints in tasks with
chained sources, because network barriers are not ordered with queued events
Key: FLINK-41547
URL: https://issues.apache.org/jira/browse/FLINK-41547
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Reporter: Roman Khachatryan
_(similar to FLINK-41442 but older and narrower; inferred from code analysis)_
*Problem*
When a task has a chained FLIP-27 source and network inputs
(MultipleInputStreamTask), an operator event such as AddSplitEvent can be
enqueued in the mailbox before a checkpoint barrier, and the chained source can
still be snapshotted before the event is applied. The enumerator's checkpoint
already records the split as assigned, while the reader's state doesn't contain
it. On restore, the split is never read again, so data is lost.
The coordinator assumes that an event is part of the next checkpoint once the
event is enqueued in the task's mailbox. OperatorCoordinatorHolder waits for
all pending events to be delivered before it completes its own checkpoint. For
a FLIP-27 source task, this holds because the checkpoint trigger is also a
mailbox mail and is ordered after the event, as long as both run in order (see
FLINK-41442). In a task with chained sources, though, the snapshot can be
triggered by a barrier from a network input, and that barrier doesn't go
through the mailbox in order with the event.
*Scenario 1: stale priority-event mail*
CheckpointedInputGate.waitForPriorityEvents enqueues a mail that drains
whatever priority events are available when the mail runs. The mail can go
stale: the input path may poll the priority event that triggered it before the
mail runs, and processPriorityEvents acknowledges this.
1. Unaligned barrier N arrives and gate mail M is enqueued. The task's input
processing polls barrier N inline, so M is stale.
2. The task thread is busy for a long time, for example on a slow record or a
long synchronous snapshot. Meanwhile checkpoint N is aborted. The coordinator
checkpoints N+1, and AddSplitEvent E is enqueued after M.
3. Barrier N+1 arrives. M runs before E, processes barrier N+1, and snapshots
the chained source without E.
*Scenario 2: barrier polled by the input processor (suspected, not verified)*
MailboxProcessor alternates between processing a batch of mails and running one
default action. If E is enqueued after createBatch(), and the default action
then polls the unaligned barrier N+1 inline from the network gate, the snapshot
runs before E. With unaligned checkpoints, the first barrier snapshots the
whole chain, including the chained sources. The window is one default-action
call, but a record that blocks, for example on output buffers under
backpressure, can widen it.
_*Notes*_
- Neither scenario was caused by FLINK-35051. Both are possible with the gate
mail as a normal mail.
- Aligned checkpoints are not affected. With chained sources, alignment also
waits for the source trigger, which is a mailbox mail ordered after the event.
- Possible directions:
- Apply pending operator events before the chained sources are snapshotted in
the barrier handler, for example by draining operator-event mails before the
snapshot.
- Or, before snapshotting the chained sources, apply the events that were
delivered before the coordinator's checkpoint.
*Related:* FLINK-41442, FLINK-35051, and the discussion in
https://github.com/apache/flink/pull/29954#discussion_r4579142169.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)