cshuo opened a new issue, #19684: URL: https://github.com/apache/hudi/issues/19684
### Task Description **What needs to be done:** - Use ZSTD as the default Parquet compression codec for Flink write clients. - Use ZSTD by default for Spark 3.5 and newer, while retaining GZIP for Spark 3.3/3.4 and Java write clients. - Preserve explicitly configured `hoodie.parquet.compression.codec` values across direct clients, DataSource/SQL, Streamer, metadata table services, and derived write configurations. - Package `zstd-jni` in the Flink bundle, while relying on the Spark runtime dependency instead of duplicating the JNI library in Spark bundles. - Document the PARQUET-2160 off-heap leak risk when Spark 3.3/3.4 Hudi file-group readers consume ZSTD Parquet files, and recommend upgrading to Spark 3.5 or newer. **Why this task is needed:** GZIP provides a good compression ratio but adds substantial CPU overhead to Parquet writes, including native Parquet log writes for MOR tables. ZSTD provides a better balance of compression throughput and storage efficiency. The default must remain version-aware because Spark 3.3/3.4 profiles use parquet-java versions whose non-vectorized reader path can still be affected by PARQUET-2160. ### Task Type Performance optimization ### Related Issues - Discussion: https://github.com/apache/hudi/discussions/19615 - Related evaluation: #14938 - Upstream leak: https://issues.apache.org/jira/browse/PARQUET-2160 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
