[
https://issues.apache.org/jira/browse/FLINK-40200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123267#comment-18123267
]
Martijn Visser commented on FLINK-40200:
----------------------------------------
Reopening, the CheckpointSyncPhaseTimeoutITCase added here failed on master,
nightly 2026-10-03 (test_cron_azure, JDK 17):
https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=79739&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3
{code}
[ERROR]
org.apache.flink.test.checkpointing.CheckpointSyncPhaseTimeoutITCase.testStuckSyncPhaseFailsJob(MiniCluster)
-- Time elapsed: 3.902 s <<< FAILURE!
org.opentest4j.AssertionFailedError:
Expecting a root cause with message:
"Task Source: Custom Source (1/1)#0 did not complete the synchronous phase of
checkpoint 1 within 50 ms."
but message was:
"Task Sink: Writer (1/1)#0 did not complete the synchronous phase of
checkpoint 1 within 50 ms.".
{code}
Both watchdogs fired, 30 ms apart. The source was blocked in
BlockingSnapshotSource.snapshotState as intended, the sink was still in its
synchronous phase:
{code}
06:43:53,605 [checkpoint 1 sync phase watchdog for Sink: Writer (1/1)#0_0] WARN
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl [] -
Task Sink: Writer (1/1)#0 did not complete the synchronous phase of checkpoint
1 within 50 ms.
[email protected]/java.lang.invoke.InvokerBytecodeGenerator.clinit(InvokerBytecodeGenerator.java:370)
...
app//org.apache.flink.streaming.runtime.tasks.OperatorChain.sendAcknowledgeCheckpointEvent(OperatorChain.java:1030)
app//org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:201)
{code}
The JobManager failed the job on the sink's failure and cancelled the source.
It is the first failure in 56 master nightlies on Azure since the test was
added. [~efrat19] could you take a look?
> Introduce sync phase timeout for TaskManagers locking up in
> org.rocksdb.Checkpoint.createCheckpoint
> ---------------------------------------------------------------------------------------------------
>
> Key: FLINK-40200
> URL: https://issues.apache.org/jira/browse/FLINK-40200
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Affects Versions: 2.3.0
> Reporter: Efrat Levitan
> Assignee: Efrat Levitan
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.4.0
>
>
> With TMs blocked by rocksdb native calls, jobs might hang for
> \{execution.checkpointing.timeout} X
> (\{execution.checkpointing.tolerable-failed-checkpoints} + 1) which can
> result in days without progress before JM issues a job restart and cancels
> the blocked task
> We experience this internally and constantly need to bump TM pods while this
> could be otherwise handled automatically by setting a lower timeout on the
> sync phase part of the checkpoint
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
