nsivabalan commented on code in PR #13007:
URL: https://github.com/apache/hudi/pull/13007#discussion_r2017646734
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -230,10 +230,11 @@ public boolean commitStats(String instantTime,
List<HoodieWriteStat> stats,
LOG.info("Committing " + instantTime + " action " + commitActionType);
// Create a Hoodie table which encapsulated the commits and files visible
HoodieTable table = createTable(config);
- HoodieCommitMetadata metadata = CommitUtils.buildMetadata(stats,
partitionToReplaceFileIds,
+ HoodieCommitMetadata oriMetadata = CommitUtils.buildMetadata(stats,
partitionToReplaceFileIds,
extraMetadata, operationType, config.getWriteSchema(),
commitActionType);
HoodieInstant inflightInstant =
table.getMetaClient().createNewInstant(State.INFLIGHT, commitActionType,
instantTime);
HeartbeatUtils.abortIfHeartbeatExpired(instantTime, table,
heartbeatClient, config);
+ HoodieCommitMetadata metadata = reconcileMetadata(table, commitActionType,
instantTime, oriMetadata);
Review Comment:
yes, you have a good point.
here is the challenging part.
1. Not all actions need to do reconcile. we only need it for delta commit
(which will create log files). So, we did not fix all other code paths.
2. We have auto commit enabled and disbaled flow. Even just for delta
commits, we have two callers and we could not bring them into a single call.
bcoz, in case of auto commit disabled flow, we prepare the commit metadata
within BaseHoodieWriteClient.commitStats and then do marker reconciliation and
then call reconcile CommitMetadata.
But incase of auto commit enabled flow, its slightly diff. we prepare the
commit metadata upfront. and then within BaseCommitActionExecutor.commit, we
call marker reconciliation, and then call appendMetadataForMissingFiles.
So, these are the two code paths that needs to call reconcileCommitMetadata.
w/ the DAG rewrite work, we plan to deprecate the auto commit enabled flow.
Something to keep in mind
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
