cshuo opened a new pull request, #19685: URL: https://github.com/apache/hudi/pull/19685
### Describe the issue this Pull Request addresses Closes #19684. Flink and Spark 3.5+ currently default Parquet base and native log files to GZIP, which provides a good compression ratio but adds significant CPU overhead to the write path. ZSTD offers a better balance of compression throughput and storage efficiency. Spark 3.3/3.4 require a compatibility exception because Hudi's non-vectorized file-group reader can still encounter the PARQUET-2160 off-heap leak when reading ZSTD files. This implements the engine- and runtime-specific behavior discussed in https://github.com/apache/hudi/discussions/19615. ### Summary and Changelog - Set the storage-level Parquet codec default to ZSTD while deferring materialization until the write engine is known. - Default Flink and Spark 3.5+ writes to ZSTD; retain GZIP for Spark 3.3/3.4, Java clients, and Spark version detection failures. - Preserve explicit `hoodie.parquet.compression.codec` values in DataSource/SQL, direct clients, metadata table services, and derived configurations. - Include `zstd-jni` in the Flink bundle and continue relying on the Spark runtime dependency without adding it to Spark bundles. - Add coverage for engine defaults, explicit overrides, partial storage configurations, metadata write configs, and actual ZSTD Parquet output. ### Impact Upgrading changes the default codec for writes without an explicit codec under Flink and Spark 3.5+. Existing Parquet files remain readable and a table can contain files using different codecs. Spark 3.3/3.4 and Java write defaults remain GZIP. The Flink bundle gains the profile-compatible `zstd-jni` classes and native libraries. Spark bundles do not duplicate `zstd-jni`; supported Spark runtimes provide it through Spark's dependency set. Explicit codec configuration continues to take precedence. ### Risk Level Medium. This intentionally changes a write default and adds a native dependency to the Flink bundle. The risk is mitigated by retaining GZIP on affected Spark versions and Java clients, preserving explicit overrides, verifying Spark 3.3 and 3.5 profiles independently, testing the Flink write-client path, and writing a Parquet file whose footer reports ZSTD compression. Validation performed: - Targeted `HoodieStorageConfig`, `HoodieWriteConfig`, and Hadoop Parquet writer tests. - `TestDataSourceUtils#testSparkVersionSpecificParquetCompressionCodecDefault` with Spark 3.3 and Spark 3.5 profiles. - `TestFlinkWriteClients#testParquetCompressionCodecDefaultAndOverride` with the Flink 2.1 profile. - Dependency-tree checks for Hadoop, Java, Flink, and Spark `zstd-jni` resolution. - Bundle-content verification that the Flink bundle contains ZSTD classes/native libraries and the Spark bundle does not contain a duplicate copy. ### Documentation Update Updated the root and Spark datasource documentation, plus the configuration description, to describe the version-specific defaults and warn that Spark 3.3/3.4 Hudi file-group readers may encounter PARQUET-2160 when reading ZSTD Parquet files. Spark 3.5+ is recommended. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
