[
https://issues.apache.org/jira/browse/FLINK-40788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18140786#comment-18140786
]
Martijn Visser commented on FLINK-40788:
----------------------------------------
One more, on release-2.4:
https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=80875&view=logs&j=320b3f94-fe78-5d7f-e965-9db30de9270e
I've dug into this. Since 2026-08-01 it failed in 10 of 198 Azure Java 11
nightly runs, and in none of the 1471 runs on other JDKs or the 162 GHA Java 11
runs. The first OOM is always Metaspace, in the first restore while the tasks
finish. The heap OOMs only follow when that attempt dies after checkpoint 3,
which is more than ten times bigger than checkpoint 1.
Locally on Java 11 the TaskManagers peak at 61-64 MB committed metaspace, also
when they get 96m, so the 64m the test sets since FLINK-13983 sits right at the
natural peak. Java 17 peaks 6-8 MB lower. It is not a classloader leak: the
job's classloader holds one class and is gone after the job. The space goes to
Flink's own classes and to lambda and reflection classes generated when 20
tasks per TaskManager start at the same time.
We should raise taskmanager.memory.jvm-metaspace.size to 96m and
taskmanager.memory.process.size to 1056m, which keeps the heap at 440 MiB.
> 'Heavy deployment end-to-end test' fails because all TaskManagers run out of
> memory on Java 11
> ----------------------------------------------------------------------------------------------
>
> Key: FLINK-40788
> URL: https://issues.apache.org/jira/browse/FLINK-40788
> Project: Flink
> Issue Type: Bug
> Components: Tests
> Affects Versions: 2.4.0
> Reporter: Martijn Visser
> Assignee: Martijn Visser
> Priority: Major
> Labels: test-stability
>
> The heavy deployment end-to-end test failed in the Java 11 nightly on master:
> https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=79325&view=logs&j=320b3f94-fe78-5d7f-e965-9db30de9270e
> {code}
> Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by
> FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3,
> backoffTimeMS=0)
> ...
> Caused by:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Could not acquire the minimum required resources.
> [FAIL] 'Heavy deployment end-to-end test' failed after 5 minutes and 12
> seconds! Test exited with exit code 1
> {code}
> All five TaskManagers logged an OutOfMemoryError during deployment, three
> "Java heap space" and two "Metaspace", before the job gave up. FLINK-16456
> had the same symptom on Java 11 in 2020.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
