wangmingzhou1986 commented on issue #9795:
URL: https://github.com/apache/paimon/issues/9795#issuecomment-5658576025
Follow-up with the heap-level evidence I offered in the report. **The
retained classloaders survive forced full GCs unchanged, so this is strong
reachability, not deferred collection.**
## How the data was obtained
The official Flink image ships a JRE, so I streamed a JDK 17 into the
running TaskManager container (`tar` pod→pod, no restart, so the
already-retained classloaders were preserved) and ran `jcmd` against the live
JVM.
One gotcha worth recording for anyone repeating this: the HotSpot attach
mechanism requires the **same uid** as the target JVM. `kubectl exec` lands as
`root` while the TaskManager runs as `flink`, and attaching as root fails with
`target process 1 doesn't respond within 10500ms`. You have to `su` to the
JVM's owner first.
## `VM.classloader_stats` on a TaskManager holding 570.7 MB Metaspace
| classloader type | live instances | classes | ChunkSz |
|---|---|---|---|
| `org.apache.flink.util.ChildFirstClassLoader` | **23** | 68,461 | 50.7 MB |
| `FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader` | **23** | 0 | – |
| **`org.apache.paimon.plugin.ComponentClassLoader`** | **19** | 7,521 | 9.3
MB |
| `org.apache.paimon.shade.org.codehaus.janino.ByteArrayClassLoader` | 55 |
55 | – |
| `jdk.internal.reflect.DelegatingClassLoader` | 564 | 564 | 0.3 MB |
| `org.codehaus.commons.compiler.util.reflect.ByteArrayClassLoader` | 13 |
13 | – |
| *(total)* | 703 CLDs | 93,193 | 71.6 MB |
`SafetyNetWrapperClassLoader` is created 1:1 with a job's user-code
classloader, and there are exactly **23** of them alongside **23**
`ChildFirstClassLoader` — i.e. 23 finished jobs whose classloaders are still
alive.
## Forced full GC — the decisive test
Two consecutive `jcmd <pid> GC.run` (both reported `Command executed
successfully`), 20 s apart:
| | before GC | after 1st | after 2nd |
|---|---|---|---|
| `ChildFirstClassLoader` | **23** | **23** | **23** |
| `SafetyNetWrapperClassLoader` | **23** | **23** | **23** |
| **`ComponentClassLoader` (Paimon)** | **19** | **19** | **19** |
| shaded-janino `ByteArrayClassLoader` | 55 | 41 | 35 |
| total CLDs | 703 | 412 | 406 |
| **Metaspace (Flink metric)** | **570.7 MB** | – | **570.7 MB** |
GC clearly ran and clearly collected things — total CLDs dropped 703 → 406
and the janino loaders shrank 55 → 35. But the **user classloaders and the
Paimon `ComponentClassLoader`s did not move at all**, and Metaspace did not
drop by a single byte.
This rules out "the JVM just hasn't collected yet" and confirms a
strong-reference retention.
## Two concrete suspects
**1. `ComponentClassLoader` is rooted at the platform loader, not at a job.**
`jcmd VM.classloaders` shows the 19 instances hanging **directly off
`PlatformClassLoader`**, siblings of Flink's own
`PluginLoader$PluginClassLoader`:
```
+-- "platform", jdk.internal.loader.ClassLoaders$PlatformClassLoader
|
+-- org.apache.paimon.plugin.ComponentClassLoader
+-- org.apache.flink.core.plugin.PluginLoader$PluginClassLoader
+-- org.apache.paimon.plugin.ComponentClassLoader
+-- org.apache.paimon.plugin.ComponentClassLoader
... (19 in total)
```
They are not children of any user classloader, they accumulate roughly 1:1
with jobs (19 vs 23), and they survive full GC — so something static is holding
them. If a `ComponentClassLoader` (or anything it loaded) also references the
job's `ChildFirstClassLoader`, that alone would pin the whole chain.
**2. The shaded janino reflection cache holds `Class` objects from the user
loader.**
In `GC.class_histogram`,
`org.apache.paimon.shade.org.codehaus.janino.ReflectionIClass$ReflectionIMethod`
appears as **eight separate entries** (997 instances ×6, 938 ×2, …). A class
name appearing multiple times in a histogram means it was **loaded once per
classloader and all copies are still live**. Janino keeps static
reflection/`IClass` caches, which is a textbook classloader pin when the cached
`Class` objects come from a child loader.
## One number I cannot yet explain
`VM.classloader_stats` accounts for **71.6 MB** of chunks in total, while
the `Metaspace` MXBean reports **570.7 MB** used on the same JVM at the same
moment. I'm reporting the discrepancy rather than explaining it away — possibly
freed-but-unreturned chunks, or a different accounting boundary. It does not
affect the conclusion above (counts, not bytes, are what stayed flat across GC).
## Environment recap
Flink 1.20.5 session cluster, Temurin 17.0.19, G1, TM
`-XX:MaxMetaspaceSize=768m`.
Workload: repeated short batch jobs, each shipping `paimon-flink-1.20`
(1.2.0/1.3.1) as a **user jar** via `sql-client -j`, doing `CREATE CATALOG ...
'type'='paimon'` on `s3://` + `SELECT COUNT(*)` against a small primary-key
table. ~18.2 MB Metaspace per finished job, never returned.
I can still produce a full heap dump with dominator/GC-root analysis if that
would be more useful than the above — just say the word. Happy to test a patch
as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
