rionmonster opened a new pull request, #29942:
URL: https://github.com/apache/flink/pull/29942

   ## What is the purpose of the change
   
   FLINK-37670 made `WatermarkAlignmentEvent` loss tolerant and keeps sending 
it after the enumerator has no more splits. When such an event does not reach 
its task because the task is no longer running, `SubtaskGatewayImpl` skips the 
task failover, but the event's future is still tracked as a failed pending 
event. A coordinator checkpoint that is waiting for pending events at that 
moment fails with "Failing OperatorCoordinator checkpoint because some 
OperatorEvents before this checkpoint barrier were not received by the target 
tasks".
   
   This happens when a source subtask finishes while an alignment announcement 
is in flight to it. In streaming it costs a failed checkpoint. It also affects 
the coordinator snapshots of batch job recovery, which are taken when an 
execution finishes.
   
   With this change, a loss that the gateway already tolerates no longer fails 
the checkpoint.
   
   ## Brief change log
   
     - `SubtaskGatewayImpl` tracks a future for the checkpoint that treats a 
loss tolerant event failing with `TaskNotRunningException` as delivered. This 
is the same condition that already skips the failover. The future returned to 
the caller still completes exceptionally.
     - `ExecutionSubtaskAccess` fails the events that are still in flight when 
the task reaches a terminal state with `TaskNotRunningException` instead of a 
plain `FlinkException`, so that this case is recognized as well.
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
     - 
`OperatorCoordinatorHolderTest#testCheckpointSucceedsIfLossTolerantEventFindsTaskNotRunning`
 and `#testBatchSnapshotSucceedsIfLossTolerantEventFindsTaskNotRunning`: a 
pending loss tolerant event fails with `TaskNotRunningException` while a 
checkpoint (or a batch snapshot) waits for it. Both fail without the change in 
`SubtaskGatewayImpl`.
     - 
`OperatorCoordinatorSchedulerTest#testCheckpointSucceedsIfTaskFinishesBeforeLossTolerantEventArrives`:
 the event is still in flight when the task finishes. It fails without the 
change in `ExecutionSubtaskAccess`.
     - 
`OperatorCoordinatorHolderTest#testCheckpointFailsIfEventFindsTaskNotRunning`: 
an event that is not loss tolerant still fails the checkpoint.
     - Manually verified with a bounded job that takes one coordinator snapshot 
right after its sources finish, using a source with watermark alignment and a 1 
ms update interval: 15 of 300 runs failed before the change, 0 of 3000 after. 
With only the `SubtaskGatewayImpl` change, 4 of 1000 runs still failed, which 
is what the `ExecutionSubtaskAccess` change covers.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes (coordinator 
checkpoints)
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Claude Code 2.1.325


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to