ahmedabu98 commented on code in PR #39250:
URL: https://github.com/apache/beam/pull/39250#discussion_r3553977476
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriterManager.java:
##########
@@ -202,7 +202,8 @@ private RecordWriter createWriter(PartitionKey
partitionKey) {
table,
icebergDestination.getFileFormat(),
filePrefix + "_" + stateToken + "_" + recordIndex,
- partitionKey);
+ partitionKey,
+ catalogConfig.getConfigProperties());
Review Comment:
`IcebergCatalogConfig`'s configProperties refers to Hadoop config
properties. We use it to initialize the Hadoop Configuration
[here](https://github.com/apache/beam/blob/dac472292a935148c31b890f59ae4b0d87695964/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergCatalogConfig.java#L96-L99)
(some old-school catalogs require it)
I think we should add a new `writeProperties` option to the [top-level
configuration](https://github.com/apache/beam/blob/dac472292a935148c31b890f59ae4b0d87695964/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergWriteSchemaTransformProvider.java#L76)
and pipe it through to IcebergIO and the RecordWriterManager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
