clairemcginty commented on PR #39645: URL: https://github.com/apache/beam/pull/39645#issuecomment-5208445558
@ahmed dug into this a bit more by writing a quick [test](https://github.com/apache/beam/pull/39645/changes/1f5573e9acf47ba64cd57375371df543fcdd8c67) to replicate the desired behavior using ManagedIO. A config like this succeeds in propagating all the desired properties on dynamic table creation: ```java Managed .write(Managed.ICEBERG) .withConfig(ImmutableMap.of( "table", "mytable", "catalog_properties", ImmutableMap.of("type", "hadoop", "warehouse", warehouse.location), "table_properties", ImmutableMap.of( "write.data.path", "mypath", "write.parquet.bloom-filter-enabled.column.mycolumn", "true"), "sort_fields", Collections.singletonList("str desc"), "partition_fields", Collections.singletonList("int"))) ``` But there isn't currently a way to do accomplish the same behavior using IcebergIO: ```java IcebergIO .writeRows(IcebergCatalogConfig.builder() .setCatalogProperties(ImmutableMap.of("...")) .setConfigProperties(ImmutableMap.of("...")) // Can't set table properties here, it's a Map<String, String> not Map<String, Object> .build() ) .to(tableId) .withPartitionFields(Collections.singletonList("int")) .withSortOrder(Collections.singletonList("str desc")) .withWriteProperties(ImmutableMap.of("...")) // Doesn't work, only propagates Parquet-level props ``` So, idk; I wonder if for simplicity's sake we should just remove `.withWriteProperties` entirely and recommend that the user switch over to ManagedIO if they need more complexity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
