the-other-tim-brown commented on code in PR #824:
URL: https://github.com/apache/incubator-xtable/pull/824#discussion_r3368205274
##########
xtable-core/src/main/java/org/apache/xtable/hudi/HudiDataFileExtractor.java:
##########
@@ -306,13 +324,68 @@ private AddedAndRemovedFiles getUpdatesToPartition(
// if a new base file was added, then the previous base file
for the group needs
// to be removed
filesToRemove.add(buildFileWithoutStats(partitionValues,
baseFile));
+ removeEmitted = true;
break;
}
}
+ // Recover old file path from timeline when CLEAN deleted it
before this sync.
+ if (newBaseFileAdded && !removeEmitted) {
+ String fileId = fileGroup.getFileGroupId().getFileId();
+ String prevCommit = fileIdToPrevCommit.get(fileId);
+ if (prevCommit != null) {
+ recoverRemovedFile(timeline, partitionPath, fileId,
prevCommit, partitionValues)
+ .ifPresent(filesToRemove::add);
+ }
+ }
});
return
AddedAndRemovedFiles.builder().added(filesToAdd).removed(filesToRemove).build();
}
+ /**
+ * Looks up the previous base file path from the timeline when CLEAN has
already deleted it from
+ * storage, so the downstream format can emit a proper REMOVE for the stale
file.
+ *
+ * @return the old {@link InternalDataFile}, or empty if the previous commit
is no longer in the
+ * visible timeline (e.g. archived before this sync ran)
+ */
+ Optional<InternalDataFile> recoverRemovedFile(
Review Comment:
This will parse the clean metadata per file. Can we create some lightweight
object that maintains an in-memory cache of the clean plans for this sync so it
is more performant when this case is hit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
