GitHub user wombatu-kun added a comment to the discussion: Use ZSTD as the default compression codec for Parquet files
On "have others switched?" - Delta and Spark default to snappy, Iceberg and Paimon to zstd; nobody stayed on gzip. Iceberg's switch is worth a look though: since 1.4.0 the zstd default is only stamped into the properties of newly created tables, so existing tables keep writing gzip on a library upgrade ([iceberg#15236](https://github.com/apache/iceberg/issues/15236)). So one thing to decide explicitly: does the new default apply to every table on the next commit after a jar upgrade, or only to new ones? Hudi persists nothing about the codec in `HoodieTableConfig` today, so "new tables only" is not free. On overriding to `snappy` for the lower Spark profiles - that does not mitigate the leak. PARQUET-2160 is on the decompression path (`CodecFactory.HeapBytesDecompressor`), so what matters is which parquet-java version *reads* a zstd file, not which one wrote it. A Spark 3.3 job reading a table that a Spark 3.5 or Flink writer filled with zstd hits the same leak, and a file stays zstd once written. Only the read-side workaround protects anyone - and the profiles that would carry it are the least covered ones: `bot.yml` runs only `spark3.5` and `spark4.2` today. On scope: `hoodie.parquet.compression.codec` is a single knob shared by base files and native Parquet log blocks (`HoodieParquetDataBlock`, `HoodieFlinkParquetDataBlock`, `RowDataInlineLogWriteHandle` and `HoodieInlineLogAppendHandle` all pass `writeConfig.getParquetCompressionCodec()`). So this means either a new log-file-specific config or flipping the shared default for base files too - different changes, and the base-file case is the one #13568 and #14938 already argued with numbers. Two smaller notes: `zstd-jni` reaches us only transitively via `parquet-hadoop` and is not in the Flink bundle's shade list; and `hoodie.logfile.to.parquet.compression.ratio` (0.35) is calibrated against today's codec, so file sizing drifts if log files go zstd while base files stay gzip. GitHub link: https://github.com/apache/hudi/discussions/19615#discussioncomment-17996901 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
