GitHub user cshuo added a comment to the discussion: Use ZSTD as the default compression codec for Parquet files
> On "have others switched?" - Delta and Spark default to snappy, Iceberg and > Paimon to zstd; nobody stayed on gzip. Iceberg's switch is worth a look > though: since 1.4.0 the zstd default is only stamped into the properties of > newly created tables, so existing tables keep writing gzip on a library > upgrade (https://github.com/apache/iceberg/issues/15236). So one thing to > decide explicitly: does the new default apply to every table on the next > commit after a jar upgrade, or only to new ones? Hudi persists nothing about > the codec in HoodieTableConfig today, so "new tables only" is not free. I would lean toward applying the new default to all tables after upgrading the Hudi version, here are some thoughts: * the compression codec is a Parquet file-level property recorded in each file’s metadata, so mixing files written with different codecs does not introduce read/write compatibility issues. * most production deployments already use ZSTD or explicitly configure another codec instead of relying on the GZIP default, so changing the default would not actually affect either of these scenarios. cc @vinothchandar @danny0405 @yihua WDYT? > On overriding to snappy for the lower Spark profiles - that does not mitigate > the leak. PARQUET-2160 is on the decompression path > (CodecFactory.HeapBytesDecompressor), so what matters is which parquet-java > version reads a zstd file, not which one wrote it. A Spark 3.3 job reading a > table that a Spark 3.5 or Flink writer filled with zstd hits the same leak, > and a file stays zstd once written. Only the read-side workaround protects > anyone - and the profiles that would carry it are the least covered ones: > bot.yml runs only spark3.5 and spark4.2 today. Agree, For affected older Spark/Parquet profiles, we should apply a reader-side workaround similar to [Iceberg’s PARQUET-2160 fix](https://github.com/apache/iceberg/pull/5681), rather than changing their writer codec. GitHub link: https://github.com/apache/hudi/discussions/19615#discussioncomment-17999879 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
