stream2000 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1218852843
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -298,6 +298,22 @@ protected void syncHoodieTable(String tableName, boolean
useRealtimeInputFormat,
LOG.info("Sync complete for " + tableName);
}
+ private boolean needToSyncAllPartitions(Option<String> lastCommitTimeSynced)
{
+ if (!lastCommitTimeSynced.isPresent()) {
+ return true;
+ }
+ if (config.getBoolean(META_SYNC_USE_STATE_TRANSIENT_TIME)) {
+ // If we use state transient time to sync partitions and the last commit
time synced is before latest archive time
+ // We need to fall back to list all partitions instead of load the whole
archive timeline
+ Option<String> latestArchiveTime = syncClient.getLastArchiveTime();
Review Comment:
@danny0405 Hi Danny, syncing both instant time and completion time to HMS
will not help solve the archive issue. Assume we have an instant completed a
long time after its instant time(for example 1 day), the archiver will archive
it immediately. And if we just looking the active timelime to find the 'hollow'
instants we will miss the instant just have been archived. To solve this
problem we still need to know the max completion time in archive timeline
anyway. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
