PDGGK opened a new pull request, #10974:
URL: https://github.com/apache/paimon/pull/10974

   ### Purpose
   
   Fixes #9795
   
   Parquet's `CodecFactory.getCodec` creates codecs with Hadoop's 
`ReflectionUtils.newInstance`. That method stores the constructor of every 
class it creates in the static `CONSTRUCTOR_CACHE`, which has strong class keys 
and is never cleared. In a Flink session cluster, Hadoop usually comes from 
`/opt/flink/lib` while Paimon comes with the job jar. The cached keys are then 
Parquet's codec classes loaded by the job's class loader, for example 
`ZstandardCodec` for the default `file.compression = zstd`. As a result, every 
finished job that reads or writes Parquet keeps its class loader, and its 
Metaspace, reachable until the TaskManager restarts. The heap dump in #9795 
shows this chain.
   
   This is the change to the codec instantiation path asked for in the review 
of #9827. Upgrading Parquet does not help, because 1.16.0 still calls 
`ReflectionUtils.newInstance`.
   
   The change adds `ParquetCodecFactory`, a `CodecFactory` subclass.
   - Its `getCodec` instantiates the codec directly: it calls the no-arg 
constructor, then `Configurable.setConf`. This is what 
`ReflectionUtils.newInstance` does for a codec.
   - Everything else is unchanged, including the shared `CODEC_BY_NAME` cache 
and its key.
   
   Paimon creates a codec factory in two places, and both now use the new class:
   - the vendored `ParquetWriter.Builder#build`;
   - `ParquetUtil#getParquetReadOptionsBuilder`, which every Parquet reader 
goes through.
   
   To check the fix, I reproduced the leak locally:
   - Hadoop client 3.3.6 is on the shared class path.
   - Each "job" runs in its own thread, with its own child class loader that 
holds the Paimon jars.
   - The job writes and reads one Parquet file through `FileFormat`.
   - The job's loader is then dropped and GC runs.
   
   Results for 10 jobs with zstd (the default):
   
   | | job class loaders reachable after GC | Metaspace growth after GC | 
`ZstandardCodec` keys in `CONSTRUCTOR_CACHE` |
   |---|---|---|---|
   | paimon-bundle 2.1.0 | 10 of 10 | +130.4 MB | 10 |
   | master (d85253bbd) | 10 of 10 | +130.5 MB | 10 |
   | this PR | 0 of 10 | +3.2 MB | 0 |
   
   This PR does not cover `snappy` or `lz4_raw`. With those codecs, Parquet's 
heap compressor and decompressor also return `SnappyCompressor` / 
`Lz4RawCompressor` (and the matching decompressors) to Hadoop's static 
`CodecPool`. That pool is keyed by these job-loaded classes. With this change, 
those codecs no longer reach `CONSTRUCTOR_CACHE`, but the same reproduction 
still retains 3 of 3 job loaders through `CodecPool`. Removing that path 
changes how compressors are reused, so it needs its own decision and is not 
part of this PR.
   
   ### Tests
   
   `ParquetCodecFactoryTest` writes and reads a Parquet file through 
`FileFormat` with `zstd`, `snappy` and `lz4_raw`. After the write and again 
after the read, it checks that Hadoop's `ReflectionUtils.CONSTRUCTOR_CACHE` 
(read through reflection) does not contain the codec class.
   - Before each case, the test removes the class from that cache and empties 
`CODEC_BY_NAME`, so codecs created by earlier tests in the same JVM cannot hide 
a regression.
   - With both call sites reverted to `new CodecFactory(...)`, all three cases 
fail at the first check, for example with 
`{org.apache.parquet.hadoop.codec.ZstandardCodec=public 
org.apache.parquet.hadoop.codec.ZstandardCodec()}` in the cache.
   - Reverting only the writer fails the check after the write. Reverting only 
the reader fails the check after the read.
   
   Ran locally on JDK 17:
   - `ParquetCodecFactoryTest`;
   - every test class under `org.apache.paimon.format.parquet` in 
`paimon-format`: 325 run, 0 failures, 1 skipped by an existing `@Disabled`;
   - checkstyle and spotless for `paimon-format`.
   
   Other modules and JDK versions are left to CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to