danny0405 commented on code in PR #19685:
URL: https://github.com/apache/hudi/pull/19685#discussion_r3881529042
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1429,6 +1433,40 @@ public static HoodieWriteConfig.Builder newBuilder() {
return new Builder();
}
+ @VisibleForTesting
+ static String getDefaultParquetCompressionCodec(EngineType engineType) {
+ switch (engineType) {
+ case FLINK:
+ return ZSTD_COMPRESSION_CODEC;
+ case SPARK:
+ // Spark 3.5 and newer use ZSTD. Spark 3.3 and 3.4 retain GZIP because
their
+ // non-vectorized file-group reader uses parquet-java 1.12.x and can
leak off-heap
+ // memory when reading ZSTD files:
https://issues.apache.org/jira/browse/PARQUET-2160.
+ Option<String> sparkVersion = getSparkRuntimeVersion();
+ return sparkVersion.isPresent()
+ && StringUtils.compareVersions(sparkVersion.get(),
MIN_SPARK_VERSION_WITH_ZSTD_DEFAULT) >= 0
+ ? ZSTD_COMPRESSION_CODEC : GZIP_COMPRESSION_CODEC;
Review Comment:
we can continue the discussion, it seems no other lake formats uses gzip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
