GitHub user cshuo added a comment to the discussion: Use ZSTD as the default compression codec for Parquet files
> Can we just control this cleanly based on parquet write version alone. @vinothchandar The leak occurs on the reader side, and ZSTD files written by newer versions may still be read by affected older profiles. But if you want use this change as a catalyst for engine upgrades, I think we can first control the default codec based on the writer profile. If community users later have a real need to read ZSTD files with affected older engine versions, we can introduce a scoped workaround at that point. WDYT? BTW, the minimum parquet-java version used by the Flink profiles is 1.13.1, which contains the upstream fix, so Flink is not affected by the memory leak issue. GitHub link: https://github.com/apache/hudi/discussions/19615#discussioncomment-18045856 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
