JingsongLi commented on code in PR #7715:
URL: https://github.com/apache/paimon/pull/7715#discussion_r3932261979
##########
paimon-core/src/main/java/org/apache/paimon/operation/OrphanFilesClean.java:
##########
@@ -243,22 +243,26 @@ protected boolean isManagedBlobPack(Path path) {
return
path.getName().endsWith(ManagedBlobReferenceFile.MANAGED_BLOB_SUFFIX);
}
- protected Set<Snapshot> safelyGetAllSnapshots(String branch) throws
IOException {
+ protected Set<Snapshot> safelyGetLiveSnapshots(String branch) throws
IOException {
FileStoreTable branchTable = table.switchToBranch(branch);
- SnapshotManager snapshotManager = branchTable.snapshotManager();
- ChangelogManager changelogManager = branchTable.changelogManager();
- TagManager tagManager = branchTable.tagManager();
- Set<Snapshot> readSnapshots = new
HashSet<>(snapshotManager.safelyGetAllSnapshots());
- readSnapshots.addAll(tagManager.taggedSnapshots());
- readSnapshots.addAll(changelogManager.safelyGetAllChangelogs());
- return readSnapshots;
+ return new
HashSet<>(branchTable.snapshotManager().safelyGetAllSnapshots());
Review Comment:
[P1] Detect an incomplete live-snapshot enumeration before deleting files
This helper still delegates to `safelyGetAllSnapshots()`, which first
freezes the snapshot-path list and then silently skips any path that disappears
while it is read. For example, it can list only S1, then a concurrent commit
creates S2 and retention-1 expiration deletes S1; the method returns an empty
set, never scans S2, and never sets `missingManifest` because no `Snapshot`
reaches the new guarded manifest readers. Local/Flink/Spark can then proceed
with no live files from this branch and delete old manifests/data files still
inherited by S2. This is the original data-loss interleaving one level earlier
than the new sentinel. Please make live enumeration itself complete (for
example, anchor/revalidate a latest snapshot and abort on a changed/incomplete
view) and add a deterministic test where commit+expiration occurs between path
listing and snapshot reading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
