danny0405 commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r758999785
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -200,20 +200,19 @@ public boolean archiveIfRequired(HoodieEngineContext
context) throws IOException
.collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
HoodieInstant.getComparableAction(i.getAction()))));
- // If metadata table is enabled, do not archive instants which are more
recent that the latest synced
- // instant on the metadata table. This is required for metadata table sync.
+ // If metadata table is enabled, do not archive instants which are more
recent that the last compaction on the
+ // metadata table.
if (config.isMetadataTableEnabled()) {
try (HoodieTableMetadata tableMetadata =
HoodieTableMetadata.create(table.getContext(), config.getMetadataConfig(),
config.getBasePath(),
FileSystemViewStorageConfig.SPILLABLE_DIR.defaultValue())) {
- Option<String> lastSyncedInstantTime = tableMetadata.getUpdateTime();
-
- if (lastSyncedInstantTime.isPresent()) {
- LOG.info("Limiting archiving of instants to last synced instant on
metadata table at " + lastSyncedInstantTime.get());
- instants = instants.filter(i ->
HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
- lastSyncedInstantTime.get()));
- } else {
- LOG.info("Not archiving as there is no instants yet on the metadata
table");
+ Option<String> latestCompactionTime =
tableMetadata.getLatestCompactionTime();
Review comment:
> This validation requires us to check each tX.deltacommmit against
corresponding instants on the dataset timeline
This is true but i did not see any reason why the delta commits on the
timeline can not archive, somehow, the decision on whether a delta commit is
committed successfully should not rely on whether it is archived, in other
words, the archiving of delta commits should be transparent to the view callers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
