zhang-arvin opened a new pull request, #9827: URL: https://github.com/apache/paimon/pull/9827
## Purpose Fixes #9795 ## Root cause Every finished Flink batch job shipping the Paimon connector as a user jar leaks ~18.2 MB TaskManager Metaspace permanently. The retained object is the job's `ChildFirstClassLoader` class for `org.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstandardCodec`, pinned by Hadoop's `ReflectionUtils.CONSTRUCTOR_CACHE` (static, strong keys) in the shared `AppClassLoader`. Paimon shades parquet (1.15.2 on release-1.3) but deliberately does not shade Hadoop, so parquet's `CodecFactory` hands the job-classloader-loaded codec class into the shared Hadoop utility that never evicts production-side. ## Fix Parquet 1.16.0 rewrites `CodecFactory` to instantiate codecs directly (`DirectCodecFactory`) instead of going through `ReflectionUtils.newInstance`, so the strong-keyed shared cache is no longer on the path. The job classloader becomes garbage-collectible after job completion. main/2.1-SNAPSHOT already runs 1.16.0; this backports the version to release-1.3. ## Verification - Bytecode check of 1.15.2 `CodecFactory`: invokes `ReflectionUtils.newInstance` → shared cache. 1.16.0 `CodecFactory` constructor builds `DirectCodecFactory` directly — no ReflectionUtils reference. - Link check: same vendored parquet sources + shade config compile on main with 1.16.0 (already proven upstream). ## Reporter's verification method (from issue thread) Heap-dump count of user-classloader keys in `CONSTRUCTOR_CACHE` should drop to 0 after upgrade. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
