yihua commented on code in PR #13007:
URL: https://github.com/apache/hudi/pull/13007#discussion_r2017645326
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -94,6 +98,21 @@ public boolean commit(String instantTime,
JavaRDD<WriteStatus> writeStatuses, Op
return commitStats(instantTime, writeStats, extraMetadata,
commitActionType, partitionToReplacedFileIds, extraPreCommitFunc);
}
+ @Override
+ protected HoodieCommitMetadata reconcileMetadata(HoodieTable table, String
commitActionType, String instantTime, HoodieCommitMetadata oriMetadata) {
+ try {
+ if
(table.getMetaClient().getTableConfig().getTableVersion().greaterThanOrEquals(HoodieTableVersion.EIGHT))
{
+ // reconciliation not required for table version 8 and above.
+ return oriMetadata;
+ }
+ return CommitMetadataUtils.reconcileMetadataForMissingFiles(table,
commitActionType, instantTime, oriMetadata,
+ config, context, storageConf, this.getClass().getSimpleName());
+ } catch (IOException e) {
+ throw new HoodieCommitException("Failed to fix commit metadata for
spurious log files "
+ + config.getBasePath() + " at time " + instantTime, e);
+ }
+ }
+
Review Comment:
Looks like this is part of marker handling. Could we fold the logic into
`WriteMarkers`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
