[ 
https://issues.apache.org/jira/browse/FLINK-41505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Nowojski closed FLINK-41505.
----------------------------------
    Resolution: Fixed

merged to master as 
[{{a128465}}|https://github.com/apache/flink/commit/a128465b145c1bfd6518cf84492f4af0a77e5484]

> Loss-tolerant OperatorEvent that hits a finished task fails the coordinator 
> checkpoint
> --------------------------------------------------------------------------------------
>
>                 Key: FLINK-41505
>                 URL: https://issues.apache.org/jira/browse/FLINK-41505
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Coordination
>    Affects Versions: 2.3.0, 2.2.1, 2.1.3
>            Reporter: Rion Williams
>            Assignee: Rion Williams
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.5.0
>
>
> h3. Problem
> FLINK-37670 made {{WatermarkAlignmentEvent}} loss tolerant 
> ({{OperatorEvent#isLossTolerant}}) and has the {{SourceCoordinator}} keep 
> sending it after the enumerator has no more splits. 
> {{SubtaskGatewayImpl#sendEvent}} accordingly skips the task failover when 
> such an event fails with {{TaskNotRunningException}}.
> However, the future of that event is still registered in the 
> {{IncompleteFuturesTracker}} and completes exceptionally. If a coordinator 
> checkpoint is waiting for pending events at that moment, 
> {{OperatorCoordinatorHolder#completeCheckpointOnceEventsAreDone}} fails it:
> {noformat}
> Coordinator checkpoint <id> for coordinator <op> is awaiting 1 pending events
> ...
> org.apache.flink.util.FlinkException: Failing OperatorCoordinator checkpoint 
> because some OperatorEvents before this checkpoint barrier were not received 
> by the target tasks.
> {noformat}
> So an event loss that the gateway treats as harmless still fails the 
> checkpoint. There are two ways to get there:
>  # The delivery fails with {{TaskNotRunningException}}, either on the 
> JobManager ({{Execution#sendOperatorEvent}}) or on the TaskManager. The 
> gateway skips the failover, but the tracked future fails.
>  # The event is still in flight when the task reaches a terminal state. 
> {{ExecutionSubtaskAccess}} then fails all pending event futures with a plain 
> {{FlinkException("Task is no longer running")}}, which is not recognized as 
> the task not running.
> h3. When it happens
> Watermark alignment is enabled, a source subtask finishes while an alignment 
> announcement (every {{updateInterval}}) is in flight to it, and a coordinator 
> checkpoint is taken before that delivery fails. This affects jobs with 
> bounded or finishing sources that checkpoint after tasks have finished: the 
> checkpoint fails and counts towards the tolerable checkpoint failures.
> h3. Reproduction
> Both paths reproduce deterministically in unit tests.
> Path 1, in {{OperatorCoordinatorHolderTest}}. It fails on master at the last 
> assertion:
> {code:java}
> @Test
> void testCheckpointSucceedsIfLossTolerantEventFindsTaskNotRunning() throws 
> Exception {
>     CompletableFuture<Acknowledge> eventSendingResult = new 
> CompletableFuture<>();
>     final EventReceivingTasks tasks =
>             
> EventReceivingTasks.createForRunningTasksWithRpcResult(eventSendingResult);
>     final OperatorCoordinatorHolder holder =
>             createCoordinatorHolder(tasks, TestingOperatorCoordinator::new);
>     // Send one loss-tolerant event without finishing it.
>     CompletableFuture<Acknowledge> sendResult =
>             getCoordinator(holder).getSubtaskGateway(0).sendEvent(new 
> TestOperatorEvent(0, true));
>     // Trigger one checkpoint.
>     CompletableFuture<byte[]> checkpointResult = new CompletableFuture<>();
>     holder.checkpointCoordinator(1, checkpointResult);
>     getCoordinator(holder).getLastTriggeredCheckpoint().complete(new byte[0]);
>     // The target task finished before the event reached it.
>     eventSendingResult.completeExceptionally(
>             new FlinkException(new TaskNotRunningException("test")));
>     assertThat(sendResult).isCompletedExceptionally();
>     assertThat(checkpointResult).isCompleted(); // fails: completed 
> exceptionally
> }
> {code}
> Path 2, in {{OperatorCoordinatorSchedulerTest}}: deploy the tasks with a 
> {{TaskExecutorOperatorEventGateway}} that never completes, send a 
> loss-tolerant event, trigger a coordinator checkpoint, then switch the task 
> to {{FINISHED}}. The checkpoint future completes exceptionally instead of 
> completing.
> h3. Proposed fix
> Two small changes:
>  # In {{SubtaskGatewayImpl#sendEvent}}, track a future for the checkpoint 
> that treats a loss-tolerant event failing with {{TaskNotRunningException}} as 
> delivered, i.e. the same condition that already skips the failover. The 
> future returned to the caller still completes exceptionally, so 
> {{SubtaskGatewayImplTest#optionalEventsIgnoreTaskNotRunning}} is unchanged.
>  # In {{ExecutionSubtaskAccess}}, fail the events that are still in flight 
> when the task reaches a terminal state with {{TaskNotRunningException}} 
> instead of a plain {{FlinkException}}, so that path is recognized as well.
> Tests: the two reproductions above and a test that an event that is not loss 
> tolerant still fails the checkpoint. I have a patch for this available in 
> https://github.com/apache/flink/pull/29942



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to