[
https://issues.apache.org/jira/browse/FLINK-41064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137207#comment-18137207
]
Purushottam Sinha commented on FLINK-41064:
-------------------------------------------
Cause: when the job is cancelled mid-checkpoint, the {{AsyncOperations}} thread
can log at {{AsyncCheckpointRunnable.java:150}} after the TaskManager has
already closed the job classloader. log4j 2.24.3 loads stack-frame classes
through the thread's context classloader, which is that closed classloader, and
doesn't catch the resulting {{IllegalStateException}}
({{ThrowableProxyHelper.java:213}}). The error goes to stderr, so the {{.out}}
check fails. log4j 2.25.3 catches it; release-2.3 got 2.25.3 through
FLINK-39146, and master is on 2.26.1.
Backport to release-2.2: https://github.com/apache/flink/pull/29767. A local
reproduction that forces this ordering with release-2.2's classloader and
TaskManager log4j config fails the {{.out}} check 10/10 on 2.24.3, with the
same trace as the CI run, and passes 10/10 on 2.25.3.
> 'State TTL Heap backend end-to-end test' fails on a "Trying to access closed
> classloader" log4j error in the TaskManager .out files
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-41064
> URL: https://issues.apache.org/jira/browse/FLINK-41064
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 2.2.2
> Environment:
> https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=80344&view=logs&j=fb37c667-81b7-5c22-dd91-846535e99a97
> (release-2.2, nightly 2026-10-21, e2e_2_cron_adaptive_scheduler)
> {code}
> The TTL verification logic failed. See *.out file for more information.
> [FAIL] 'State TTL Heap backend end-to-end test' failed after 1 minutes and 5
> seconds! Test exited with exit code 1
> {code}
> The test fails when any .out file is non-empty. All three TaskManager .out
> files contain only this log4j error, written at 04:42:32.46, about 0.4 s
> after the job was cancelled:
> {code}
> 2026-10-21T04:42:32.460222966Z AsyncOperations-thread-1 ERROR An exception
> occurred processing Appender MainAppender java.lang.IllegalStateException:
> Trying to access closed classloader. Please check if you store classloaders
> directly or indirectly in static fields. ...
> at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:189)
> at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:197)
> ...
> at
> org.apache.logging.log4j.core.impl.ThrowableProxyHelper.toExtendedStackTrace(ThrowableProxyHelper.java:112)
> ...
> at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:150)
> {code}
> The PrintFailedVerifications sink received 0 records on all three subtasks,
> so no TTL verification failed.
> Reporter: Martijn Visser
> Priority: Major
> Labels: test-stability
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
