parisni commented on code in PR #10841:
URL: https://github.com/apache/hudi/pull/10841#discussion_r1519531770
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -689,28 +692,49 @@ public void createDatabase(String databaseName) {
public Option<String> getLastCommitTimeSynced(String tableName) {
try {
Table table = getTable(awsGlue, databaseName, tableName);
- return
Option.ofNullable(table.parameters().get(HOODIE_LAST_COMMIT_TIME_SYNC));
+ return
Option.ofNullable(table.parameters().getOrDefault(HOODIE_LAST_COMMIT_TIME_SYNC,
null));
} catch (Exception e) {
throw new HoodieGlueSyncException("Fail to get last sync commit time for
" + tableId(databaseName, tableName), e);
}
}
+ @Override
+ public Option<String> getLastCommitCompletionTimeSynced(String tableName) {
+ // Get the last commit completion time from the TBLproperties
+ try {
+ Table table = getTable(awsGlue, databaseName, tableName);
+ return
Option.ofNullable(table.parameters().getOrDefault(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC,
null));
+ } catch (Exception e) {
+ throw new HoodieHiveSyncException("Failed to get the last commit
completion time synced from the table " + tableName, e);
+ }
+ }
+
@Override
public void close() {
awsGlue.close();
}
@Override
public void updateLastCommitTimeSynced(String tableName) {
- if (!getActiveTimeline().lastInstant().isPresent()) {
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<String> lastCommitSynced =
activeTimeline.lastInstant().map(HoodieInstant::getTimestamp);
+ Option<String> lastCommitCompletionSynced = activeTimeline
+ .getInstantsOrderedByCompletionTime()
+ .skip(activeTimeline.countInstants() - 1)
+ .findFirst()
+ .map(i -> Option.of(i.getCompletionTime()))
+ .orElse(Option.empty());
+ if (lastCommitSynced.isPresent()) {
+ try {
+ HashMap<String, String> propertyMap = new HashMap<>();
+ propertyMap.put(HOODIE_LAST_COMMIT_TIME_SYNC, lastCommitSynced.get());
+ propertyMap.put(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC,
lastCommitCompletionSynced.get());
Review Comment:
I just copied the hive sync code from
https://github.com/apache/hudi/blob/ff0e67f78dfde76cc7bc02ce9b9c11cba846197d/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java#L364-L363
BTW I agree an will propose a fix for both in later commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
