PDGGK opened a new pull request, #10974:
URL: https://github.com/apache/paimon/pull/10974
### Purpose
Fixes #9795
Parquet's `CodecFactory.getCodec` creates codecs with Hadoop's
`ReflectionUtils.newInstance`. That method stores the constructor of every
class it creates in the static `CONSTRUCTOR_CACHE`, which has strong class keys
and is never cleared. In a Flink session cluster, Hadoop usually comes from
`/opt/flink/lib` while Paimon comes with the job jar. The cached keys are then
Parquet's codec classes loaded by the job's class loader, for example
`ZstandardCodec` for the default `file.compression = zstd`. As a result, every
finished job that reads or writes Parquet keeps its class loader, and its
Metaspace, reachable until the TaskManager restarts. The heap dump in #9795
shows this chain.
This is the change to the codec instantiation path asked for in the review
of #9827. Upgrading Parquet does not help, because 1.16.0 still calls
`ReflectionUtils.newInstance`.
The change adds `ParquetCodecFactory`, a `CodecFactory` subclass.
- Its `getCodec` instantiates the codec directly: it calls the no-arg
constructor, then `Configurable.setConf`. This is what
`ReflectionUtils.newInstance` does for a codec.
- Everything else is unchanged, including the shared `CODEC_BY_NAME` cache
and its key.
Paimon creates a codec factory in two places, and both now use the new class:
- the vendored `ParquetWriter.Builder#build`;
- `ParquetUtil#getParquetReadOptionsBuilder`, which every Parquet reader
goes through.
To check the fix, I reproduced the leak locally:
- Hadoop client 3.3.6 is on the shared class path.
- Each "job" runs in its own thread, with its own child class loader that
holds the Paimon jars.
- The job writes and reads one Parquet file through `FileFormat`.
- The job's loader is then dropped and GC runs.
Results for 10 jobs with zstd (the default):
| | job class loaders reachable after GC | Metaspace growth after GC |
`ZstandardCodec` keys in `CONSTRUCTOR_CACHE` |
|---|---|---|---|
| paimon-bundle 2.1.0 | 10 of 10 | +130.4 MB | 10 |
| master (d85253bbd) | 10 of 10 | +130.5 MB | 10 |
| this PR | 0 of 10 | +3.2 MB | 0 |
This PR does not cover `snappy` or `lz4_raw`. With those codecs, Parquet's
heap compressor and decompressor also return `SnappyCompressor` /
`Lz4RawCompressor` (and the matching decompressors) to Hadoop's static
`CodecPool`. That pool is keyed by these job-loaded classes. With this change,
those codecs no longer reach `CONSTRUCTOR_CACHE`, but the same reproduction
still retains 3 of 3 job loaders through `CodecPool`. Removing that path
changes how compressors are reused, so it needs its own decision and is not
part of this PR.
### Tests
`ParquetCodecFactoryTest` writes and reads a Parquet file through
`FileFormat` with `zstd`, `snappy` and `lz4_raw`. After the write and again
after the read, it checks that Hadoop's `ReflectionUtils.CONSTRUCTOR_CACHE`
(read through reflection) does not contain the codec class.
- Before each case, the test removes the class from that cache and empties
`CODEC_BY_NAME`, so codecs created by earlier tests in the same JVM cannot hide
a regression.
- With both call sites reverted to `new CodecFactory(...)`, all three cases
fail at the first check, for example with
`{org.apache.parquet.hadoop.codec.ZstandardCodec=public
org.apache.parquet.hadoop.codec.ZstandardCodec()}` in the cache.
- Reverting only the writer fails the check after the write. Reverting only
the reader fails the check after the read.
Ran locally on JDK 17:
- `ParquetCodecFactoryTest`;
- every test class under `org.apache.paimon.format.parquet` in
`paimon-format`: 325 run, 0 failures, 1 skipped by an existing `@Disabled`;
- checkstyle and spotless for `paimon-format`.
Other modules and JDK versions are left to CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]