[
https://issues.apache.org/jira/browse/FLINK-40804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18143410#comment-18143410
]
Gunjan Aggarwal edited comment on FLINK-40804 at 11/12/26, 4:01 AM:
--------------------------------------------------------------------
Hi [~martijnvisser]
I have added the RCA doc of the test case.
[^RCA FLINK-40804 — UnalignedCheckpointRescaleWithMixedExchangesITCase (Test
case.pdf]
Brief:
*Root cause:* The Source 2 → Map FORWARD edge is aligned-only, so the barrier
can’t overtake queued data. When the first checkpoint fires late, job 1 saves
much more in-flight data, mainly in Co-Map’s output to Map after keyBy. Job 2
rescales Map after keyBy to parallelism 1, where {{Thread.sleep(5)}} caps it at
~200 rec/s. Restoring that data backpressures Co-Map, Map and Source 2. In one
run, a median of ~11k records sat ahead of the barrier while Map read only ~19
rec/s, about 593 s against the 600 s timeout. The failed runs waited 600–745 s.
Map after keyBy = 1 alone is not enough: with small in-flight data, 0/30 runs
failed.
*Fix:* PR [apache/flink#27688|https://github.com/apache/flink/pull/27688] on
master switches the record type from Long to a 100-char String. The queue is
bounded in buffers, not records, so far fewer records fit per buffer and need
to drain (66/100 → 0/100 in our loop). The PR also fixes
MAX_RETAINED_CHECKPOINTS (set on StreamExecutionEnvironment, but read from
cluster config), disables CUSTOM_PARTITIONER in
{{UnalignedCheckpointRescaleITCase}} (FLINK-39162), adds multi-round rescaling,
and names the unspilling thread with task/subtask. On release-1.20 only the
MAX_RETAINED commit is backported, and it does not stop the failure (Martijn’s
2026-10-27 run). The Long → String change still needs backporting
was (Author: JIRAUSER315066):
Hi [~martijnvisser]
I have added the RCA doc of the test case.
Brief:
*Root cause:* The Source 2 → Map FORWARD edge is aligned-only, so the barrier
can’t overtake queued data. When the first checkpoint fires late, job 1 saves
much more in-flight data, mainly in Co-Map’s output to Map after keyBy. Job 2
rescales Map after keyBy to parallelism 1, where {{Thread.sleep(5)}} caps it at
~200 rec/s. Restoring that data backpressures Co-Map, Map and Source 2. In one
run, a median of ~11k records sat ahead of the barrier while Map read only ~19
rec/s, about 593 s against the 600 s timeout. The failed runs waited 600–745 s.
Map after keyBy = 1 alone is not enough: with small in-flight data, 0/30 runs
failed.
*Fix:* PR [apache/flink#27688|https://github.com/apache/flink/pull/27688] on
master switches the record type from Long to a 100-char String. The queue is
bounded in buffers, not records, so far fewer records fit per buffer and need
to drain (66/100 → 0/100 in our loop). The PR also fixes
MAX_RETAINED_CHECKPOINTS (set on StreamExecutionEnvironment, but read from
cluster config), disables CUSTOM_PARTITIONER in
{{UnalignedCheckpointRescaleITCase}} (FLINK-39162), adds multi-round rescaling,
and names the unspilling thread with task/subtask. On release-1.20 only the
MAX_RETAINED commit is backported, and it does not stop the failure (Martijn’s
2026-10-27 run). The Long → String change still needs backporting
> UnalignedCheckpointRescaleWithMixedExchangesITCase.testRescaleFromUnalignedCheckpoint
> fails with "Exceeded checkpoint tolerable failure threshold"
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40804
> URL: https://issues.apache.org/jira/browse/FLINK-40804
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.20.6
> Reporter: Martijn Visser
> Assignee: Gunjan Aggarwal
> Priority: Major
> Attachments: RCA FLINK-40804 —
> UnalignedCheckpointRescaleWithMixedExchangesITCase (Test case.pdf
>
>
> https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=79252&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3
> (release-1.20 nightly, 2026-09-19, test_cron_jdk17 tests)
> {code}
> [ERROR]
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleWithMixedExchangesITCase.testRescaleFromUnalignedCheckpoint[Test
> case 3] -- Time elapsed: 633.5 s <<< ERROR!
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> NoRestartBackoffTimeStrategy
> ...
> Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint
> tolerable failure threshold. The latest checkpoint failed due to Checkpoint
> expired before completing., view the Checkpoint History tab or the Job
> Manager log to find out why continuous checkpoints failed.
> {code}
> The rescaled job restored from checkpoint 1, its source stayed in
> INITIALIZING for 26 seconds, checkpoints 2 to 6 were declined with "tasks not
> ready", and checkpoint 7 expired after the 10 minute timeout. The other four
> test cases passed in the same run.
> This is not the tempdir race from FLINK-40446, which was backported on
> 2026-09-16.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
