jerqi commented on code in PR #8313:
URL: https://github.com/apache/iceberg/pull/8313#discussion_r1311009860
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java:
##########
@@ -418,4 +424,96 @@ public String branch() {
return branch;
}
+
+ public String parquetCompressionCodec() {
+ return confParser
+ .stringConf()
+ .option(SparkWriteOptions.COMPRESSION_CODEC)
+ .sessionConf(SparkSQLProperties.COMPRESSION_CODEC)
+ .tableProperty(TableProperties.PARQUET_COMPRESSION)
+ .defaultValue(TableProperties.PARQUET_COMPRESSION_DEFAULT)
+ .parse();
+ }
+
+ public String parquetCompressionLevel() {
+ return confParser
+ .stringConf()
+ .option(SparkWriteOptions.COMPRESSION_LEVEL)
+ .sessionConf(SparkSQLProperties.COMPRESSION_LEVEL)
+ .tableProperty(TableProperties.PARQUET_COMPRESSION_LEVEL)
+ .defaultValue(TableProperties.PARQUET_COMPRESSION_LEVEL_DEFAULT)
+ .parseOptional();
+ }
+
+ public String avroCompressionCodec() {
+ return confParser
+ .stringConf()
+ .option(SparkWriteOptions.COMPRESSION_CODEC)
+ .sessionConf(SparkSQLProperties.COMPRESSION_CODEC)
+ .tableProperty(TableProperties.AVRO_COMPRESSION)
+ .defaultValue(TableProperties.AVRO_COMPRESSION_DEFAULT)
+ .parse();
+ }
+
+ public String avroCompressionLevel() {
+ return confParser
+ .stringConf()
+ .option(SparkWriteOptions.COMPRESSION_LEVEL)
+ .sessionConf(SparkSQLProperties.COMPRESSION_LEVEL)
+ .tableProperty(TableProperties.AVRO_COMPRESSION_LEVEL)
+ .defaultValue(TableProperties.AVRO_COMPRESSION_LEVEL_DEFAULT)
+ .parseOptional();
+ }
+
+ public String orcCompressionCodec() {
+ return confParser
+ .stringConf()
+ .option(SparkWriteOptions.COMPRESSION_CODEC)
+ .sessionConf(SparkSQLProperties.COMPRESSION_CODEC)
+ .tableProperty(TableProperties.ORC_COMPRESSION)
+ .defaultValue(TableProperties.ORC_COMPRESSION_DEFAULT)
+ .parse();
+ }
+
+ public String orcCompressionStrategy() {
+ return confParser
+ .stringConf()
+ .option(SparkWriteOptions.COMPRESSION_STRATEGY)
+ .sessionConf(SparkSQLProperties.COMPRESSION_STRATEGY)
+ .tableProperty(TableProperties.ORC_COMPRESSION_STRATEGY)
+ .defaultValue(TableProperties.ORC_COMPRESSION_STRATEGY_DEFAULT)
+ .parse();
+ }
+
+ public Map<String, String> writeProperties(FileFormat format) {
+ Map<String, String> writeProperties = Maps.newHashMap();
+
+ switch (format) {
+ case PARQUET:
+ writeProperties.put(PARQUET_COMPRESSION, parquetCompressionCodec());
+ String parquetCompressionLevel = parquetCompressionLevel();
+ if (parquetCompressionLevel != null) {
+ writeProperties.put(PARQUET_COMPRESSION_LEVEL,
parquetCompressionLevel);
Review Comment:
Sorry for this mistake. I will check how to valid the correctness of the
case.
<img width="683" alt="企业微信截图_9057e533-5033-48a3-817e-a22238277d8d"
src="https://github.com/apache/iceberg/assets/8159038/a77f21dc-fbed-4351-b38d-c9a05792a037";>
If we don't set the delete compression codec, we will reuse data compression
codec. So the test case passed.
Could I raise a pr https://github.com/apache/iceberg/pull/8438 to fix this
issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
