cshuo commented on code in PR #19685:
URL: https://github.com/apache/hudi/pull/19685#discussion_r3880629079
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1429,6 +1433,40 @@ public static HoodieWriteConfig.Builder newBuilder() {
return new Builder();
}
+ @VisibleForTesting
+ static String getDefaultParquetCompressionCodec(EngineType engineType) {
+ switch (engineType) {
+ case FLINK:
+ return ZSTD_COMPRESSION_CODEC;
+ case SPARK:
+ // Spark 3.5 and newer use ZSTD. Spark 3.3 and 3.4 retain GZIP because
their
+ // non-vectorized file-group reader uses parquet-java 1.12.x and can
leak off-heap
+ // memory when reading ZSTD files:
https://issues.apache.org/jira/browse/PARQUET-2160.
+ Option<String> sparkVersion = getSparkRuntimeVersion();
+ return sparkVersion.isPresent()
+ && StringUtils.compareVersions(sparkVersion.get(),
MIN_SPARK_VERSION_WITH_ZSTD_DEFAULT) >= 0
+ ? ZSTD_COMPRESSION_CODEC : GZIP_COMPRESSION_CODEC;
Review Comment:
Snappy’s lower compression ratio can significantly increase base-file size
and storage costs, especially on object storage like S3. This concern was also
discussed in
[#13568](https://github.com/apache/hudi/issues/13568#issuecomment-3085987801),
so GZIP is retained as the default codec for older Spark versions. cc
@vinothchandar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
