yihua commented on code in PR #7921:
URL: https://github.com/apache/hudi/pull/7921#discussion_r1103472610
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1279,8 +1279,21 @@ protected void tryUpgrade(HoodieTableMetaClient
metaClient, Option<String> insta
}
}
+ /**
+ * Rolls back the failed delta commits corresponding to the indexing action.
+ * <p>
+ * TODO(HUDI-5733): This should be cleaned up once the proper fix of
rollbacks
+ * in the metadata table is landed.
+ *
+ * @return {@code true} if rollback happens; {@code false} otherwise.
+ */
+ public boolean lazyRollbackFailedIndexing() {
+ return tableServiceClient.lazyRollbackFailedIndexing();
Review Comment:
This API has to be added so that we can reuse part of rollback logic inside
`BaseHoodieTableServiceClient`.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -699,6 +740,19 @@ protected List<String>
getInstantsToRollback(HoodieTableMetaClient metaClient, H
Stream<HoodieInstant> inflightInstantsStream =
getInflightTimelineExcludeCompactionAndClustering(metaClient)
.getReverseOrderedInstants();
if (cleaningPolicy.isEager()) {
+ // Metadata table uses eager cleaning policy, but we need to exclude
inflight delta commits
+ // from the async indexer (`HoodieIndexer`).
+ // TODO(HUDI-5733): This should be cleaned up once the proper fix of
rollbacks in the
+ // metadata table is landed.
+ if
(HoodieTableMetadata.isMetadataTable(metaClient.getBasePathV2().toString())) {
+ return
inflightInstantsStream.map(HoodieInstant::getTimestamp).filter(entry -> {
+ if (curInstantTime.isPresent()) {
+ return !entry.equals(curInstantTime.get());
+ } else {
+ return !isDeltaCommitFromIndexing(entry);
Review Comment:
This is to exclude the inflight delta commits due to async indexing from the
eager rollback.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1069,6 +1073,7 @@ protected void cleanIfNecessary(BaseHoodieWriteClient
writeClient, String instan
// delta commits synced over will not have an instant time lesser than the
last completed instant on the
// metadata table.
writeClient.clean(instantTime + "002");
+ writeClient.lazyRollbackFailedIndexing();
Review Comment:
Lazy rollback of inflight delta commits from async indexing only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
