[
https://issues.apache.org/jira/browse/FLINK-41505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rion Williams updated FLINK-41505:
----------------------------------
Description:
h3. Problem
FLINK-37670 made {{WatermarkAlignmentEvent}} loss tolerant
({{OperatorEvent#isLossTolerant}}) and has the {{SourceCoordinator}} keep
sending it after the enumerator has no more splits.
{{SubtaskGatewayImpl#sendEvent}} accordingly skips the task failover when such
an event fails with {{TaskNotRunningException}}.
However, the future of that event is still registered in the
{{IncompleteFuturesTracker}} and completes exceptionally. If a coordinator
checkpoint is waiting for pending events at that moment,
{{OperatorCoordinatorHolder#completeCheckpointOnceEventsAreDone}} fails it:
{noformat}
Coordinator checkpoint <id> for coordinator <op> is awaiting 1 pending events
...
org.apache.flink.util.FlinkException: Failing OperatorCoordinator checkpoint
because some OperatorEvents before this checkpoint barrier were not received by
the target tasks.
{noformat}
So an event loss that the gateway treats as harmless still fails the
checkpoint. There are two ways to get there:
# The delivery fails with {{TaskNotRunningException}}, either on the
JobManager ({{Execution#sendOperatorEvent}}) or on the TaskManager. The gateway
skips the failover, but the tracked future fails.
# The event is still in flight when the task reaches a terminal state.
{{ExecutionSubtaskAccess}} then fails all pending event futures with a plain
{{FlinkException("Task is no longer running")}}, which is not recognized as the
task not running.
h3. When it happens
Watermark alignment is enabled, a source subtask finishes while an alignment
announcement (every {{updateInterval}}) is in flight to it, and a coordinator
checkpoint is taken before that delivery fails. This affects jobs with bounded
or finishing sources that checkpoint after tasks have finished: the checkpoint
fails and counts towards the tolerable checkpoint failures.
h3. Reproduction
Both paths reproduce deterministically in unit tests.
Path 1, in {{OperatorCoordinatorHolderTest}}. It fails on master at the last
assertion:
{code:java}
@Test
void testCheckpointSucceedsIfLossTolerantEventFindsTaskNotRunning() throws
Exception {
CompletableFuture<Acknowledge> eventSendingResult = new
CompletableFuture<>();
final EventReceivingTasks tasks =
EventReceivingTasks.createForRunningTasksWithRpcResult(eventSendingResult);
final OperatorCoordinatorHolder holder =
createCoordinatorHolder(tasks, TestingOperatorCoordinator::new);
// Send one loss-tolerant event without finishing it.
CompletableFuture<Acknowledge> sendResult =
getCoordinator(holder).getSubtaskGateway(0).sendEvent(new
TestOperatorEvent(0, true));
// Trigger one checkpoint.
CompletableFuture<byte[]> checkpointResult = new CompletableFuture<>();
holder.checkpointCoordinator(1, checkpointResult);
getCoordinator(holder).getLastTriggeredCheckpoint().complete(new byte[0]);
// The target task finished before the event reached it.
eventSendingResult.completeExceptionally(
new FlinkException(new TaskNotRunningException("test")));
assertThat(sendResult).isCompletedExceptionally();
assertThat(checkpointResult).isCompleted(); // fails: completed
exceptionally
}
{code}
Path 2, in {{OperatorCoordinatorSchedulerTest}}: deploy the tasks with a
{{TaskExecutorOperatorEventGateway}} that never completes, send a loss-tolerant
event, trigger a coordinator checkpoint, then switch the task to {{FINISHED}}.
The checkpoint future completes exceptionally instead of completing.
h3. Proposed fix
Two small changes:
# In {{SubtaskGatewayImpl#sendEvent}}, track a future for the checkpoint that
treats a loss-tolerant event failing with {{TaskNotRunningException}} as
delivered, i.e. the same condition that already skips the failover. The future
returned to the caller still completes exceptionally, so
{{SubtaskGatewayImplTest#optionalEventsIgnoreTaskNotRunning}} is unchanged.
# In {{ExecutionSubtaskAccess}}, fail the events that are still in flight when
the task reaches a terminal state with {{TaskNotRunningException}} instead of a
plain {{FlinkException}}, so that path is recognized as well.
Tests: the two reproductions above and a test that an event that is not loss
tolerant still fails the checkpoint. I have a patch ready.
was:
h3. Problem
FLINK-37670 made {{WatermarkAlignmentEvent}} loss tolerant
({{OperatorEvent#isLossTolerant}}) and has the {{SourceCoordinator}} keep
sending it after the enumerator has no more splits.
{{SubtaskGatewayImpl#sendEvent}} accordingly skips the task failover when such
an event fails with {{TaskNotRunningException}}.
However, the future of that event is still registered in the
{{IncompleteFuturesTracker}} and completes exceptionally. If a coordinator
checkpoint is waiting for pending events at that moment,
{{OperatorCoordinatorHolder#completeCheckpointOnceEventsAreDone}} fails it:
{noformat}
Coordinator checkpoint <id> for coordinator <op> is awaiting 1 pending events
...
org.apache.flink.util.FlinkException: Failing OperatorCoordinator checkpoint
because some OperatorEvents before this checkpoint barrier were not received by
the target tasks.
{noformat}
So an event loss that the gateway treats as harmless still fails the
checkpoint. There are two ways to get there:
# The delivery fails with {{TaskNotRunningException}}, either on the
JobManager ({{Execution#sendOperatorEvent}}) or on the TaskManager. The gateway
skips the failover, but the tracked future fails.
# The event is still in flight when the task reaches a terminal state.
{{ExecutionSubtaskAccess}} then fails all pending event futures with a plain
{{FlinkException("Task is no longer running")}}, which is not recognized as the
task not running.
h3. When it happens
Watermark alignment is enabled, a source subtask finishes while an alignment
announcement (every {{updateInterval}}) is in flight to it, and a coordinator
checkpoint is taken before that delivery fails. This affects jobs with bounded
or finishing sources that checkpoint after tasks have finished: the checkpoint
fails and counts towards the tolerable checkpoint failures.
h3. Reproduction
Both paths reproduce deterministically in unit tests.
Path 1, in {{OperatorCoordinatorHolderTest}}. It fails on master at the last
assertion:
{code:java}
@Test
void testCheckpointSucceedsIfLossTolerantEventFindsTaskNotRunning() throws
Exception {
CompletableFuture<Acknowledge> eventSendingResult = new
CompletableFuture<>();
final EventReceivingTasks tasks =
EventReceivingTasks.createForRunningTasksWithRpcResult(eventSendingResult);
final OperatorCoordinatorHolder holder =
createCoordinatorHolder(tasks, TestingOperatorCoordinator::new);
// Send one loss-tolerant event without finishing it.
CompletableFuture<Acknowledge> sendResult =
getCoordinator(holder).getSubtaskGateway(0).sendEvent(new
TestOperatorEvent(0, true));
// Trigger one checkpoint.
CompletableFuture<byte[]> checkpointResult = new CompletableFuture<>();
holder.checkpointCoordinator(1, checkpointResult);
getCoordinator(holder).getLastTriggeredCheckpoint().complete(new byte[0]);
// The target task finished before the event reached it.
eventSendingResult.completeExceptionally(
new FlinkException(new TaskNotRunningException("test")));
assertThat(sendResult).isCompletedExceptionally();
assertThat(checkpointResult).isCompleted(); // fails: completed
exceptionally
}
{code}
Path 2, in {{OperatorCoordinatorSchedulerTest}}: deploy the tasks with a
{{TaskExecutorOperatorEventGatew never completes, send a loss-tolerant event,
trigger a coordinator checkpoint, then switch the task to{{FINISHED}}. The
checkpoint future completes exceptionally instead of completing.
h3. Proposed fix
Two small changes:
# In {{SubtaskGatewayImpl#sendEvent}}, track a future for the checkpoint that
treats a loss-tolerant eveunningException}} as delivered, i.e. the same
condition that already skips the failover. The futurereturned to the caller
still completes exceptionally, so
{{SubtaskGatewayImplTest#optionalEventsIgnoreTaskNotRunning}} is unchanged.
# In {{ExecutionSubtaskAccess}}, fail the events that are still in flight when
the task reaches a terminningException}} instead of a plain {{FlinkException}},
so that path is recognized as well.
Tests: the two reproductions above and a test that an event thill fails the
checkpoint. I have a patch ready.
> Loss-tolerant OperatorEvent that hits a finished task fails the coordinator
> checkpoint
> --------------------------------------------------------------------------------------
>
> Key: FLINK-41505
> URL: https://issues.apache.org/jira/browse/FLINK-41505
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 2.3.0, 2.2.1, 2.1.3
> Reporter: Rion Williams
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.5.0
>
>
> h3. Problem
> FLINK-37670 made {{WatermarkAlignmentEvent}} loss tolerant
> ({{OperatorEvent#isLossTolerant}}) and has the {{SourceCoordinator}} keep
> sending it after the enumerator has no more splits.
> {{SubtaskGatewayImpl#sendEvent}} accordingly skips the task failover when
> such an event fails with {{TaskNotRunningException}}.
> However, the future of that event is still registered in the
> {{IncompleteFuturesTracker}} and completes exceptionally. If a coordinator
> checkpoint is waiting for pending events at that moment,
> {{OperatorCoordinatorHolder#completeCheckpointOnceEventsAreDone}} fails it:
> {noformat}
> Coordinator checkpoint <id> for coordinator <op> is awaiting 1 pending events
> ...
> org.apache.flink.util.FlinkException: Failing OperatorCoordinator checkpoint
> because some OperatorEvents before this checkpoint barrier were not received
> by the target tasks.
> {noformat}
> So an event loss that the gateway treats as harmless still fails the
> checkpoint. There are two ways to get there:
> # The delivery fails with {{TaskNotRunningException}}, either on the
> JobManager ({{Execution#sendOperatorEvent}}) or on the TaskManager. The
> gateway skips the failover, but the tracked future fails.
> # The event is still in flight when the task reaches a terminal state.
> {{ExecutionSubtaskAccess}} then fails all pending event futures with a plain
> {{FlinkException("Task is no longer running")}}, which is not recognized as
> the task not running.
> h3. When it happens
> Watermark alignment is enabled, a source subtask finishes while an alignment
> announcement (every {{updateInterval}}) is in flight to it, and a coordinator
> checkpoint is taken before that delivery fails. This affects jobs with
> bounded or finishing sources that checkpoint after tasks have finished: the
> checkpoint fails and counts towards the tolerable checkpoint failures.
> h3. Reproduction
> Both paths reproduce deterministically in unit tests.
> Path 1, in {{OperatorCoordinatorHolderTest}}. It fails on master at the last
> assertion:
> {code:java}
> @Test
> void testCheckpointSucceedsIfLossTolerantEventFindsTaskNotRunning() throws
> Exception {
> CompletableFuture<Acknowledge> eventSendingResult = new
> CompletableFuture<>();
> final EventReceivingTasks tasks =
>
> EventReceivingTasks.createForRunningTasksWithRpcResult(eventSendingResult);
> final OperatorCoordinatorHolder holder =
> createCoordinatorHolder(tasks, TestingOperatorCoordinator::new);
> // Send one loss-tolerant event without finishing it.
> CompletableFuture<Acknowledge> sendResult =
> getCoordinator(holder).getSubtaskGateway(0).sendEvent(new
> TestOperatorEvent(0, true));
> // Trigger one checkpoint.
> CompletableFuture<byte[]> checkpointResult = new CompletableFuture<>();
> holder.checkpointCoordinator(1, checkpointResult);
> getCoordinator(holder).getLastTriggeredCheckpoint().complete(new byte[0]);
> // The target task finished before the event reached it.
> eventSendingResult.completeExceptionally(
> new FlinkException(new TaskNotRunningException("test")));
> assertThat(sendResult).isCompletedExceptionally();
> assertThat(checkpointResult).isCompleted(); // fails: completed
> exceptionally
> }
> {code}
> Path 2, in {{OperatorCoordinatorSchedulerTest}}: deploy the tasks with a
> {{TaskExecutorOperatorEventGateway}} that never completes, send a
> loss-tolerant event, trigger a coordinator checkpoint, then switch the task
> to {{FINISHED}}. The checkpoint future completes exceptionally instead of
> completing.
> h3. Proposed fix
> Two small changes:
> # In {{SubtaskGatewayImpl#sendEvent}}, track a future for the checkpoint
> that treats a loss-tolerant event failing with {{TaskNotRunningException}} as
> delivered, i.e. the same condition that already skips the failover. The
> future returned to the caller still completes exceptionally, so
> {{SubtaskGatewayImplTest#optionalEventsIgnoreTaskNotRunning}} is unchanged.
> # In {{ExecutionSubtaskAccess}}, fail the events that are still in flight
> when the task reaches a terminal state with {{TaskNotRunningException}}
> instead of a plain {{FlinkException}}, so that path is recognized as well.
> Tests: the two reproductions above and a test that an event that is not loss
> tolerant still fails the checkpoint. I have a patch ready.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)