GitHub user cshuo added a comment to the discussion: Use ZSTD as the default compression codec for Parquet files
The memory leak occurs on the reader side, so overriding the write codec to Snappy for older Spark versions would not fully address it. Older Spark versions may still read ZSTD Parquet files written by newer engines. A more consistent approach would be to use ZSTD as the default across versions and apply a reader-side workaround for the affected older Spark/Parquet versions, similar to Iceberg’s fix before upgrading to Parquet 1.13.0: [apache/iceberg#5681](https://github.com/apache/iceberg/pull/5681). GitHub link: https://github.com/apache/hudi/discussions/19615#discussioncomment-17998750 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
