rangareddy opened a new issue, #19316:
URL: https://github.com/apache/hudi/issues/19316
**Describe the problem you faced**
`AWSGlueCatalogSyncClient.updateTableComments` no longer applies any column
or partition column comments since the AWS SDK v2 upgrade.
Its helper `setComments` builds a new `Column` and discards the result
instead of replacing the element in the list:
```java
private void setComments(List<Column> columns, Map<String, Option<String>>
commentsMap) {
columns.forEach(column -> {
String comment = commentsMap.getOrDefault(column.name(),
Option.empty()).orElse(null);
Column.builder().comment(comment).build(); // result dropped, column
is unchanged
});
}
```
AWS SDK v2 model classes are immutable, so mutating in place is not
possible. Before the SDK v2 upgrade (e9dd73fc08fd, #9347) this code called
`column.setComment(...)` on the mutable v1 model, which worked. As a result
`updateTableComments` never detects a change and always returns `false`, and no
comments are synced to Glue on the update path.
The existing Glue comment tests do not catch this because they only assert
comments that were pre-baked into the mocked `Column` objects.
**Expected behavior**
With `hoodie.datasource.hive_sync.sync_comment=true`, column and partition
column comments should be applied to the Glue table.
**Possible fix**
Rebuild the column lists with the comment applied, e.g.:
```java
for (int i = 0; i < columns.size(); i++) {
Column column = columns.get(i);
String comment = commentsMap.getOrDefault(column.name(),
Option.empty()).orElse(null);
columns.set(i, column.toBuilder().comment(comment).build());
}
```
(using mutable copies of `storageDescriptor.columns()` /
`table.partitionKeys()`, which are returned as unmodifiable lists by the SDK),
and update the tests to assert comments that were not already present on the
input columns.
**Environment Description**
* Hudi version: master (1.3.0-SNAPSHOT), broken since the AWS SDK v2 upgrade
(#9347)
**Additional context**
Found during review of #19289, which fixes the equivalent HMS paths. Related
to HUDI-8843 / #17359.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
