danny0405 commented on code in PR #13249:
URL: https://github.com/apache/hudi/pull/13249#discussion_r2103592710
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -233,19 +237,28 @@ protected void syncHoodieTable(String tableName, boolean
useRealtimeInputFormat,
LOG.info("Trying to sync hoodie table " + tableName + " with base path " +
syncClient.getBasePath()
+ " of type " + syncClient.getTableType());
- // create database if needed
- checkAndCreateDatabase();
-
final boolean tableExists = syncClient.tableExists(tableName);
- // Get the parquet schema for this table looking at the latest commit
- MessageType schema =
syncClient.getStorageSchema(!config.getBoolean(HIVE_SYNC_OMIT_METADATA_FIELDS));
// if table exists and location of the metastore table doesn't match the
hoodie base path, recreate the table
if (tableExists &&
!FSUtils.comparePathsWithoutScheme(syncClient.getBasePath(),
syncClient.getTableLocation(tableName))) {
LOG.info("basepath is updated for the table {}", tableName);
recreateAndSyncHiveTable(tableName, useRealtimeInputFormat,
readAsOptimized);
return;
}
+ // Check if any sync is required
+ if (tableExists && isIncrementalSync()) {
+ Option<String> lastCommitTimeSynced =
syncClient.getLastCommitTimeSynced(tableName);
+ Option<String> lastCommitCompletionTimeSynced =
syncClient.getLastCommitCompletionTimeSynced(tableName);
+ if (lastCommitTimeSynced.isPresent()) {
+ if (TimelineUtils.getCommitsTimelineAfter(syncClient.getMetaClient(),
lastCommitTimeSynced.get(), lastCommitCompletionTimeSynced).countInstants() ==
0) {
Review Comment:
Both of these cases are kind of long running services and the table would be
synced together with following-up successful commits, kind of think the check
is redundant though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
