rionmonster opened a new pull request, #29942:
URL: https://github.com/apache/flink/pull/29942
## What is the purpose of the change
FLINK-37670 made `WatermarkAlignmentEvent` loss tolerant and keeps sending
it after the enumerator has no more splits. When such an event does not reach
its task because the task is no longer running, `SubtaskGatewayImpl` skips the
task failover, but the event's future is still tracked as a failed pending
event. A coordinator checkpoint that is waiting for pending events at that
moment fails with "Failing OperatorCoordinator checkpoint because some
OperatorEvents before this checkpoint barrier were not received by the target
tasks".
This happens when a source subtask finishes while an alignment announcement
is in flight to it. In streaming it costs a failed checkpoint. It also affects
the coordinator snapshots of batch job recovery, which are taken when an
execution finishes.
With this change, a loss that the gateway already tolerates no longer fails
the checkpoint.
## Brief change log
- `SubtaskGatewayImpl` tracks a future for the checkpoint that treats a
loss tolerant event failing with `TaskNotRunningException` as delivered. This
is the same condition that already skips the failover. The future returned to
the caller still completes exceptionally.
- `ExecutionSubtaskAccess` fails the events that are still in flight when
the task reaches a terminal state with `TaskNotRunningException` instead of a
plain `FlinkException`, so that this case is recognized as well.
## Verifying this change
This change added tests and can be verified as follows:
-
`OperatorCoordinatorHolderTest#testCheckpointSucceedsIfLossTolerantEventFindsTaskNotRunning`
and `#testBatchSnapshotSucceedsIfLossTolerantEventFindsTaskNotRunning`: a
pending loss tolerant event fails with `TaskNotRunningException` while a
checkpoint (or a batch snapshot) waits for it. Both fail without the change in
`SubtaskGatewayImpl`.
-
`OperatorCoordinatorSchedulerTest#testCheckpointSucceedsIfTaskFinishesBeforeLossTolerantEventArrives`:
the event is still in flight when the task finishes. It fails without the
change in `ExecutionSubtaskAccess`.
-
`OperatorCoordinatorHolderTest#testCheckpointFailsIfEventFindsTaskNotRunning`:
an event that is not loss tolerant still fails the checkpoint.
- Manually verified with a bounded job that takes one coordinator snapshot
right after its sources finish, using a source with watermark alignment and a 1
ms update interval: 15 of 300 runs failed before the change, 0 of 3000 after.
With only the `SubtaskGatewayImpl` change, 4 of 1000 runs still failed, which
is what the `ExecutionSubtaskAccess` change covers.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes (coordinator
checkpoints)
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]