wangmingzhou1986 commented on issue #9795:
URL: https://github.com/apache/paimon/issues/9795#issuecomment-5659661539
I took the heap dump I offered, and **it corrects the attribution in my
original report.** Paimon is not retaining the classloader through its own
caches — but there is still a concrete, Paimon-side trigger, and I can now name
the exact retaining field.
## TL;DR
The dominant retainer is **Hadoop's `ReflectionUtils.CONSTRUCTOR_CACHE`**, a
`static final ConcurrentHashMap<Class<?>, Constructor<?>>` with **strong
keys**, living in the **shared** `AppClassLoader`
(`/opt/flink/lib/hadoop-common`). It holds the `Class` object of
**`org.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstandardCodec`**,
which is loaded by the job's `ChildFirstClassLoader`. One entry per job,
forever.
```
class org.apache.hadoop.util.ReflectionUtils <- AppClassLoader
(shared, /opt/flink/lib)
.static CONSTRUCTOR_CACHE (ConcurrentHashMap, strong keys)
-> Node.key = class
org.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstandardCodec
-> .<classloader> = org.apache.flink.util.ChildFirstClassLoader <-
pinned forever
```
Hadoop's own javadoc on that field states it *"pins the classes so they
can't be garbage collected until ReflectionUtils can be collected"*.
`clearCache()` exists but is package-private and test-only; nothing evicts in
production.
## Why this is reachable from Paimon
Paimon shades parquet but (correctly) does **not** shade Hadoop. So the
shaded `CodecFactory` hands a **user-classloader-loaded** class into the
**shared** Hadoop utility:
```java
// parquet-hadoop CodecFactory.getCodec()
codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass,
ConfigurationUtil.createHadoopConfiguration(conf));
CODEC_BY_NAME.put(codecCacheKey, codec);
```
```java
// hadoop-common ReflectionUtils.newInstance()
Constructor<T> meth = (Constructor<T>) CONSTRUCTOR_CACHE.get(theClass);
if (meth == null) { meth = theClass.getDeclaredConstructor(EMPTY_ARRAY);
...; CONSTRUCTOR_CACHE.put(theClass, meth); }
```
`CodecFactory.release()` clears `compressors`/`decompressors` only — it
touches neither `CODEC_BY_NAME` nor Hadoop's `CONSTRUCTOR_CACHE`. So **correct
usage does not release it either**; any job that writes/reads a `zstd` parquet
table permanently pins its own classloader.
## Measured breakdown on one TaskManager
19 live `ChildFirstClassLoader` instances, dumped with `jcmd GC.heap_dump`
(1.5 s, no downtime — I used a TaskManager with 0 allocated slots):
| retaining mechanism | loaders pinned |
|---|---|
| **Hadoop `ReflectionUtils.CONSTRUCTOR_CACHE` -> shaded `ZstandardCodec`**
| **9** |
| lingering threads via `Thread.inheritedAccessControlContext ->
ProtectionDomain.classloader` | 8 |
| `java.io.ClassCache$CacheRef` (SoftReference, `ObjectStreamClass` cache) |
1 |
The `ReflectionUtils` copy in `AppClassLoader` holds 11 cache entries, **9
of them keyed by user-classloader classes, all of them `ZstandardCodec`** — one
per job. (There is a second `ReflectionUtils` copy inside
`PluginLoader$PluginClassLoader` from `flink-s3-fs-hadoop`; its cache has 5
entries and pins nothing. If you reproduce this, **enumerate all copies of the
class** — I first sampled only one and got the wrong answer.)
The lingering-thread group is mostly **not** Paimon: 4 ×
`connection-pool-<mysql-host>:3306 housekeeper` (HikariCP, from the Flink CDC
MySQL source), 1 × `java-sdk-progress-listener-callback-thread` (AWS SDK), 2 ×
Flink's own `IOManagerAsync` / `FileChannelManagerImpl` shutdown hooks. I'm
reporting those to the relevant projects separately.
## A hypothesis I had, and which the heap disproves
Before dumping, I traced the source and suspected `CodeGenLoader` /
`S3Loader`:
```java
private static PluginLoader loader; // no cleanup anywhere
private static synchronized PluginLoader getLoader() { if (loader == null)
loader = new PluginLoader(...); return loader; }
```
combined with `ComponentClassLoader` holding `ownerClassLoader`. It looks
exactly like a classloader pin. **It is not one here** — neither appears on any
retention path. The static field lives in a class loaded *by the user
classloader itself*, so it cannot pin that loader from outside. I'm stating
this explicitly so nobody spends time on it: the 19 `ComponentClassLoader`
instances are a *consequence* of the retained loaders, not the cause.
## Possible fixes on the Paimon side
Since the pinned class is Paimon's own shaded class, Paimon can break the
chain without waiting for Hadoop:
1. Don't route shaded codec classes through shared `ReflectionUtils`.
`ReflectionUtils.newInstance` is just `getDeclaredConstructor().newInstance()`
plus `setConf`; instantiating directly in Paimon's shaded `CodecFactory` avoids
the shared strong-keyed cache entirely.
2. Or relocate `org.apache.hadoop.util.ReflectionUtils` into the shaded
namespace as well, so the cache lives and dies with the job classloader.
Option 1 looks like a small, contained change. I'm happy to prepare and test
a PR if you agree with the direction — I can verify with the same heap-dump
method (the count of user-loaded keys in `CONSTRUCTOR_CACHE` should go to 0).
## Method notes for anyone reproducing
- The container had no `CAP_SYS_PTRACE` and `hostPID=false`, so the
Serviceability Agent (`jhsdb revptrs`) was unusable. `jcmd GC.heap_dump` goes
through the HotSpot attach mechanism and needs no ptrace — that is the right
tool here. Attach must run as the JVM's own uid (`su flink`), not root.
- When computing retention paths, two filters are mandatory or the answer is
wrong: (a) drop `referent` edges of `java.lang.ref.Reference` subclasses
(48,977 of them here) — weak/soft/phantom refs do not retain; keep
`FinalReference`; (b) ignore `ROOT_STICKY_CLASS` self-rooting, since HotSpot
marks every loaded class as a root, which would make any live loader trivially
"reachable" through its own classes.
- One thing I cannot explain and am not going to explain away: 7 of the 19
loaders are **unreachable** after those filters, even though `GC.heap_dump`
runs a full GC first. `VM.classloader_stats` reported 26 CLDs against 19 loader
objects in the heap, so classloader-data teardown lagging object collection is
consistent with it, but I have not proven that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
