huaxingao commented on code in PR #16094:
URL: https://github.com/apache/iceberg/pull/16094#discussion_r4462778579
##########
docs/docs/configuration.md:
##########
@@ -54,6 +54,8 @@ Iceberg tables support table properties to configure table
behavior, like the de
| write.parquet.dict-size-bytes | 2097152 (2 MB)
| Parquet dictionary page size
|
| write.parquet.compression-codec | zstd
| Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed
|
| write.parquet.compression-level | null
| Parquet compression level
|
+| write.parquet.compression-codec.column.col1 | (not set)
| Per-column compression codec override for column 'col1': zstd, brotli,
lz4, gzip, snappy, uncompressed
|
+| write.parquet.compression-level.column.col1 | (not set)
| Per-column compression level override for column 'col1'. When a column
overrides the codec but not the level, 'write.parquet.compression-level' is
applied only if the overriding codec matches the table codec; otherwise the
overriding codec's default level is used |
Review Comment:
For gzip, this property takes a number (0-9), but the table-level
`write.parquet.compression-level` only accepts Hadoop enum names like
`BEST_SPEED`.
Could we mention the valid values here, so users don't copy the table-level
gzip value into the column property?
##########
parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java:
##########
@@ -774,6 +779,119 @@ private Pair<File, Long> generateFile(
return Pair.of(file, size);
}
+ @Test
+ void globalCompressionCodecAppliesToAllColumns() throws Exception {
+ File file = writeIntAndStringFields(ImmutableMap.of(PARQUET_COMPRESSION,
"snappy"), 5);
+
+ assertThat(readColumnCodecs(file))
+ .containsEntry("int_field", CompressionCodecName.SNAPPY)
+ .containsEntry("string_field", CompressionCodecName.SNAPPY);
+ }
+
+ @Test
+ void perColumnCompressionCodec() throws Exception {
+ File file =
+ writeIntAndStringFields(
+ ImmutableMap.of(
+ PARQUET_COMPRESSION,
+ "zstd",
+ PARQUET_COLUMN_COMPRESSION_CODEC_PREFIX + "int_field",
+ "snappy"),
+ 5);
+
+ assertThat(readColumnCodecs(file))
+ .containsEntry("int_field", CompressionCodecName.SNAPPY)
+ .containsEntry("string_field", CompressionCodecName.ZSTD);
+ }
+
+ @Test
+ void perColumnCompressionCodecForNonExistentColumnIsIgnored() throws
Exception {
+ File file =
+ writeIntAndStringFields(
+ ImmutableMap.of(
+ PARQUET_COMPRESSION,
+ "snappy",
+ PARQUET_COLUMN_COMPRESSION_CODEC_PREFIX + "non_existent_field",
+ "zstd"),
+ 5);
+
+ assertThat(readColumnCodecs(file))
+ .containsEntry("int_field", CompressionCodecName.SNAPPY)
+ .containsEntry("string_field", CompressionCodecName.SNAPPY);
+ }
+
+ @Test
+ void perColumnCompressionLevelWritesSuccessfully() throws Exception {
+ File file =
+ writeIntAndStringFields(
+ ImmutableMap.of(
+ PARQUET_COMPRESSION,
+ "snappy",
+ PARQUET_COLUMN_COMPRESSION_CODEC_PREFIX + "int_field",
+ "zstd",
+ PARQUET_COLUMN_COMPRESSION_LEVEL_PREFIX + "int_field",
+ "9"),
+ 5);
+
+ assertThat(readColumnCodecs(file))
Review Comment:
This test only checks the codec. Could we also check that the level is used?
For example, write the same data at zstd level 1 and level 19, and check that
the level 19 file is smaller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
