codope commented on code in PR #10841:
URL: https://github.com/apache/hudi/pull/10841#discussion_r1518792330
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -689,28 +692,49 @@ public void createDatabase(String databaseName) {
public Option<String> getLastCommitTimeSynced(String tableName) {
try {
Table table = getTable(awsGlue, databaseName, tableName);
- return
Option.ofNullable(table.parameters().get(HOODIE_LAST_COMMIT_TIME_SYNC));
+ return
Option.ofNullable(table.parameters().getOrDefault(HOODIE_LAST_COMMIT_TIME_SYNC,
null));
} catch (Exception e) {
throw new HoodieGlueSyncException("Fail to get last sync commit time for
" + tableId(databaseName, tableName), e);
}
}
+ @Override
+ public Option<String> getLastCommitCompletionTimeSynced(String tableName) {
+ // Get the last commit completion time from the TBLproperties
+ try {
+ Table table = getTable(awsGlue, databaseName, tableName);
+ return
Option.ofNullable(table.parameters().getOrDefault(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC,
null));
+ } catch (Exception e) {
+ throw new HoodieHiveSyncException("Failed to get the last commit
completion time synced from the table " + tableName, e);
+ }
+ }
+
@Override
public void close() {
awsGlue.close();
}
@Override
public void updateLastCommitTimeSynced(String tableName) {
- if (!getActiveTimeline().lastInstant().isPresent()) {
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<String> lastCommitSynced =
activeTimeline.lastInstant().map(HoodieInstant::getTimestamp);
+ Option<String> lastCommitCompletionSynced = activeTimeline
+ .getInstantsOrderedByCompletionTime()
+ .skip(activeTimeline.countInstants() - 1)
+ .findFirst()
+ .map(i -> Option.of(i.getCompletionTime()))
+ .orElse(Option.empty());
+ if (lastCommitSynced.isPresent()) {
+ try {
+ HashMap<String, String> propertyMap = new HashMap<>();
+ propertyMap.put(HOODIE_LAST_COMMIT_TIME_SYNC, lastCommitSynced.get());
+ propertyMap.put(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC,
lastCommitCompletionSynced.get());
Review Comment:
Is there a case when `lastCommitCompletionSynced` can be empty? If so, it
can throw `NoSuchElementException`. Should we also check
`lastCommitCompletionSynced.isPresent()`?
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -689,28 +692,49 @@ public void createDatabase(String databaseName) {
public Option<String> getLastCommitTimeSynced(String tableName) {
try {
Table table = getTable(awsGlue, databaseName, tableName);
- return
Option.ofNullable(table.parameters().get(HOODIE_LAST_COMMIT_TIME_SYNC));
+ return
Option.ofNullable(table.parameters().getOrDefault(HOODIE_LAST_COMMIT_TIME_SYNC,
null));
} catch (Exception e) {
throw new HoodieGlueSyncException("Fail to get last sync commit time for
" + tableId(databaseName, tableName), e);
}
}
+ @Override
+ public Option<String> getLastCommitCompletionTimeSynced(String tableName) {
Review Comment:
Where is this method called?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
