zherenyu831 commented on a change in pull request #2784:
URL: https://github.com/apache/hudi/pull/2784#discussion_r611273466
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java
##########
@@ -76,15 +76,16 @@ public static HoodieArchivedMetaEntry
createMetaWrapper(HoodieInstant hoodieInst
archivedMetaWrapper.setHoodieReplaceCommitMetadata(ReplaceArchivalHelper.convertReplaceCommitMetadata(replaceCommitMetadata));
} else if (hoodieInstant.isInflight()) {
// inflight replacecommit files have the same meta data body as
HoodieCommitMetadata
- // so we could re-use it without further creating an inflight
extension
- HoodieCommitMetadata inflightCommitMetadata = HoodieCommitMetadata
-
.fromBytes(metaClient.getActiveTimeline().getInstantDetails(hoodieInstant).get(),
HoodieCommitMetadata.class);
-
archivedMetaWrapper.setHoodieInflightReplaceMetadata(convertCommitMetadata(inflightCommitMetadata));
+ // so we could re-use it without further creating an inflight
extension.
+ // Or inflight replacecommit files are empty under clustering
circumstance
+ Option<HoodieCommitMetadata> inflightCommitMetadata =
getInflightReplaceMetadata(metaClient, hoodieInstant);
+ if (inflightCommitMetadata.isPresent()) {
+
archivedMetaWrapper.setHoodieInflightReplaceMetadata(convertCommitMetadata(inflightCommitMetadata.get()));
+ }
} else {
- // we may have some cases with empty HoodieRequestedReplaceMetadata
- // e.g. insert_overwrite_table or insert_overwrite without clustering
- Option<HoodieRequestedReplaceMetadata> requestedReplaceMetadata =
- ClusteringUtils.getRequestedReplaceMetadata(metaClient,
hoodieInstant);
+ // we may have cases with empty HoodieRequestedReplaceMetadata e.g.
insert_overwrite_table or insert_overwrite
+ // without clustering. However, we should revisit the requested
commit file standardization
+ Option<HoodieRequestedReplaceMetadata> requestedReplaceMetadata =
getRequestedReplaceMetadata(metaClient, hoodieInstant);
Review comment:
How about have a replaceUtils? I think maybe it is a good timing to
decouple logic of replace commit from clustering
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
