[
https://issues.apache.org/jira/browse/FLINK-35335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137211#comment-18137211
]
Purushottam Sinha commented on FLINK-35335:
-------------------------------------------
I looked into the recent master failures. There are two problems stacked on top
of each other.
The first failure in each run is the IllegalStateException in
{{RemoteInputChannel#convertToPriorityEvent}}. Since FLINK-39018 a blocked
{{PipelinedSubpartition}} still hands out priority buffers. That was meant for
the RECOVERY_COMPLETION block during recovery, but it also applies to the block
after an aligned barrier, and FLINK-40519 made it work for credited remote
readers too. So when checkpoint N gets aborted while its aligned barrier is
still queued downstream, the barrier of N+1 times out to unaligned and
overtakes it on the same channel. The channel only remembers the latest
barrier, so converting the announcement of N fails the checkState. In the
2026-10-05 log this happens right after checkpoints 27, 29 and 30 were aborted
because sources were finishing.
The endless loop is the old problem from FLINK-31036: the job restores from a
checkpoint taken after some subtasks finished, those get empty state, and the
test functions throw on that. With unlimited restarts it never stops. Also,
{{OnceFailingAggregator.hasFailed}} is never reset, so the region failover run
never injects its own failure.
I could reproduce it locally by looping the test with the seed of that nightly:
7 out of 24 runs hit the exception and 4 of them hung. With the fix none did.
PR: https://github.com/apache/flink/pull/29766. Priority buffers now only
bypass a RECOVERY_COMPLETION block, so other blocks behave like before
FLINK-39018, and the test disables checkpoints after tasks finish.
> StateCheckpointedITCase failed fatally with 127 exit code
> ---------------------------------------------------------
>
> Key: FLINK-35335
> URL: https://issues.apache.org/jira/browse/FLINK-35335
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.19.1, 2.2.2, 2.3.1, 2.4.0
> Reporter: Ryan Skraba
> Priority: Critical
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59499&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=8379
> {code}
> May 13 01:50:22 01:50:22.272 [INFO] Tests run: 6, Failures: 0, Errors: 0,
> Skipped: 0, Time elapsed: 30.03 s -- in
> org.apache.flink.test.streaming.runtime.CacheITCase
> May 13 01:50:23 01:50:23.142 [INFO] Tests run: 1, Failures: 0, Errors: 0,
> Skipped: 0, Time elapsed: 5.234 s -- in
> org.apache.flink.test.streaming.experimental.CollectITCase
> May 13 01:50:23 01:50:23.611 [INFO]
> May 13 01:50:23 01:50:23.611 [INFO] Results:
> May 13 01:50:23 01:50:23.611 [INFO]
> May 13 01:50:23 01:50:23.611 [WARNING] Tests run: 1960, Failures: 0, Errors:
> 0, Skipped: 25
> May 13 01:50:23 01:50:23.611 [INFO]
> May 13 01:50:23 01:50:23.674 [INFO]
> ------------------------------------------------------------------------
> May 13 01:50:23 01:50:23.674 [INFO] BUILD FAILURE
> May 13 01:50:23 01:50:23.674 [INFO]
> ------------------------------------------------------------------------
> May 13 01:50:23 01:50:23.676 [INFO] Total time: 41:24 min
> May 13 01:50:23 01:50:23.677 [INFO] Finished at: 2024-05-13T01:50:23Z
> May 13 01:50:23 01:50:23.677 [INFO]
> ------------------------------------------------------------------------
> May 13 01:50:23 01:50:23.677 [WARNING] The requested profile
> "skip-webui-build" could not be activated because it does not exist.
> May 13 01:50:23 01:50:23.678 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.2.2:test (integration-tests)
> on project flink-tests:
> May 13 01:50:23 01:50:23.678 [ERROR]
> May 13 01:50:23 01:50:23.678 [ERROR] Please refer to
> /__w/2/s/flink-tests/target/surefire-reports for the individual test results.
> May 13 01:50:23 01:50:23.678 [ERROR] Please refer to dump files (if any
> exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> May 13 01:50:23 01:50:23.678 [ERROR] ExecutionException The forked VM
> terminated without properly saying goodbye. VM crash or System.exit called?
> May 13 01:50:23 01:50:23.678 [ERROR] Command was /bin/sh -c cd
> '/__w/2/s/flink-tests' && '/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java'
> '-XX:+UseG1GC' '-Xms256m' '-XX:+IgnoreUnrecognizedVMOptions'
> '--add-opens=java.base/java.util=ALL-UNNAMED'
> '--add-opens=java.base/java.io=ALL-UNNAMED' '-Xmx1536m' '-jar'
> '/__w/2/s/flink-tests/target/surefire/surefirebooter-20240513010926195_686.jar'
> '/__w/2/s/flink-tests/target/surefire' '2024-05-13T01-09-20_665-jvmRun1'
> 'surefire-20240513010926195_684tmp' 'surefire_206-20240513010926195_685tmp'
> May 13 01:50:23 01:50:23.679 [ERROR] Error occurred in starting fork, check
> output in log
> May 13 01:50:23 01:50:23.679 [ERROR] Process Exit Code: 127
> May 13 01:50:23 01:50:23.679 [ERROR] Crashed tests:
> May 13 01:50:23 01:50:23.679 [ERROR]
> org.apache.flink.test.checkpointing.StateCheckpointedITCase
> May 13 01:50:23 01:50:23.679 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> May 13 01:50:23 01:50:23.679 [ERROR] Command was /bin/sh -c cd
> '/__w/2/s/flink-tests' && '/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java'
> '-XX:+UseG1GC' '-Xms256m' '-XX:+IgnoreUnrecognizedVMOptions'
> '--add-opens=java.base/java.util=ALL-UNNAMED'
> '--add-opens=java.base/java.io=ALL-UNNAMED' '-Xmx1536m' '-jar'
> '/__w/2/s/flink-tests/target/surefire/surefirebooter-20240513010926195_686.jar'
> '/__w/2/s/flink-tests/target/surefire' '2024-05-13T01-09-20_665-jvmRun1'
> 'surefire-20240513010926195_684tmp' 'surefire_206-20240513010926195_685tmp'
> May 13 01:50:23 01:50:23.679 [ERROR] Error occurred in starting fork, check
> output in log
> May 13 01:50:23 01:50:23.679 [ERROR] Process Exit Code: 127
> May 13 01:50:23 01:50:23.679 [ERROR] Crashed tests:
> May 13 01:50:23 01:50:23.679 [ERROR]
> org.apache.flink.test.checkpointing.StateCheckpointedITCase
> May 13 01:50:23 01:50:23.679 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:456)
> May 13 01:50:23 01:50:23.679 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:418)
> May 13 01:50:23 01:50:23.679 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:297)
> May 13 01:50:23 01:50:23.679 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
> {code}
> In the maven logs, {{runCheckpointedProgram[FailoverStrategy:
> RestartPipelinedRegionFailoverStrategy]}} is started but never completes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
