ahmedabu98 opened a new pull request, #36542: URL: https://github.com/apache/beam/pull/36542
Passing table properties to the data writers. Primary reason for this PR is that since 1.4.0, Iceberg changed its Parquet compression default from `gzip` to `zstd`. This change was reflected only in the default table properties, not the data writers. We also should be passing in table properties to the writers anyways. Confirmed from logs that the table default decompressor is now zstd Before: ``` Oct 16, 2025 11:52:29 AM org.apache.hadoop.io.compress.CodecPool getDecompressor INFO: Got brand-new decompressor [.gz] Oct 16, 2025 11:52:29 AM org.apache.hadoop.io.compress.CodecPool getDecompressor INFO: Got brand-new decompressor [.gz] Oct 16, 2025 11:52:29 AM org.apache.hadoop.io.compress.CodecPool getDecompressor INFO: Got brand-new decompressor [.gz] ``` After: ``` Oct 16, 2025 11:52:58 AM org.apache.hadoop.io.compress.CodecPool getDecompressor INFO: Got brand-new decompressor [.zstd] Oct 16, 2025 11:52:59 AM org.apache.hadoop.io.compress.CodecPool getDecompressor INFO: Got brand-new decompressor [.zstd] Oct 16, 2025 11:52:59 AM org.apache.hadoop.io.compress.CodecPool getDecompressor INFO: Got brand-new decompressor [.zstd] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
