vamsikarnika commented on code in PR #11677:
URL: https://github.com/apache/hudi/pull/11677#discussion_r1696343482
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -560,26 +560,11 @@ public void createOrReplaceTable(String tableName,
// Create a temp table will validate the schema before dropping and
recreating the table
String tempTableName = generateTempTableName(tableName);
createTable(tempTableName, storageSchema, inputFormatClass,
outputFormatClass, serdeClass, serdeProperties, tableProperties);
-
- Table tempTable = getTable(awsGlue, databaseName, tempTableName);
- final Instant now = Instant.now();
- TableInput updatedTableInput = TableInput.builder()
- .name(tableName)
- .tableType(tempTable.tableType())
- .parameters(tempTable.parameters())
- .partitionKeys(tempTable.partitionKeys())
- .storageDescriptor(tempTable.storageDescriptor())
- .lastAccessTime(now)
- .lastAnalyzedTime(now)
- .build();
-
- UpdateTableRequest request = UpdateTableRequest.builder()
- .databaseName(databaseName)
- .skipArchive(skipTableArchive)
- .tableInput(updatedTableInput)
- .build();
- awsGlue.updateTable(request).get();
+ // drop the temp table
dropTable(tempTableName);
+ // drop and recreate the actual table
+ dropTable(tableName);
+ createTable(tableName, storageSchema, inputFormatClass,
outputFormatClass, serdeClass, serdeProperties, tableProperties);
Review Comment:
Hi @CTTY, With the previous approach createOrReplaceTable is not functioning
in the expected way. We observed that partitions are not dropped when the
updateTable is called with latest properties. This is causing issues with the
glue sync.
In this approach, we're dropping the table and creating the new table which
should solve this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
