prashantwason commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r758965003
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -200,20 +200,19 @@ public boolean archiveIfRequired(HoodieEngineContext
context) throws IOException
.collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
HoodieInstant.getComparableAction(i.getAction()))));
- // If metadata table is enabled, do not archive instants which are more
recent that the latest synced
- // instant on the metadata table. This is required for metadata table sync.
+ // If metadata table is enabled, do not archive instants which are more
recent that the last compaction on the
+ // metadata table.
if (config.isMetadataTableEnabled()) {
try (HoodieTableMetadata tableMetadata =
HoodieTableMetadata.create(table.getContext(), config.getMetadataConfig(),
config.getBasePath(),
FileSystemViewStorageConfig.SPILLABLE_DIR.defaultValue())) {
- Option<String> lastSyncedInstantTime = tableMetadata.getUpdateTime();
-
- if (lastSyncedInstantTime.isPresent()) {
- LOG.info("Limiting archiving of instants to last synced instant on
metadata table at " + lastSyncedInstantTime.get());
- instants = instants.filter(i ->
HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
- lastSyncedInstantTime.get()));
- } else {
- LOG.info("Not archiving as there is no instants yet on the metadata
table");
+ Option<String> latestCompactionTime =
tableMetadata.getLatestCompactionTime();
Review comment:
In the synchronous metadata table design (0.10+), the following is the
order of how an action completes in HUDI when metadata table is enabled
(assuming a commit).
t1.commit.requested
t1.commit.inflight
<all writes complete here>
finalizeWrite()
<Write to metadata table here which creates
t1.deltacommit on metadata table>
<NOTE: the t1.commit on the dataset has not yet
completed as we have not created t1.commit yet>
t1.commit is created.
So the deltacommit to the metadata table is completed before the commit
completes. Remember, these are two different HUDI tables.
There is an error condition - when we crash / fail before the very last step
- t1.deltacommit is created on dataset but t1.commit is not created. In this
case, t1.deltacommit is NOT VALID on the metadata table. We ignore
t1.deltacommit while reading from metadata table. This validation requires us
to check each tX.deltacommmit against corresponding instants on the dataset
timeline. So if the dataset timeline is archived independent of metadata table
compaction then this logic will not work.
This is further complicated in the presence of multiple writers - each of
them in parallel writing to the dataset (e.g. t2.commit.inflight,
t3.commit.inflight, etc) and writing to the metadata table when complete. So
metadata table could have t3.deltacommit but no t2.deltacommit which could have
failed or not completed yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
