LiangRuoShui opened a new issue, #9220: URL: https://github.com/apache/paimon/issues/9220
### Search before asking - [x] I searched in the issues and found no issue covering this exact create-tag versus snapshot-expiration race. ### Paimon version - 1.3.1 - The same race is still present on master commit 54e64e364c1ef62206c173b4fec06a15320b0d32. ### Compute Engine Java API. The problem is in paimon-core and is compute-engine independent. ### Minimal reproduce step Preconditions: 1. Create several snapshots. 2. Choose an old snapshot S inside the range that will be expired. 3. Ensure a later snapshot has deleted or replaced files that are still visible in S. 4. Run tag creation and snapshot expiration concurrently. A deterministic test can inject a latch immediately after ExpireSnapshotsImpl collects tagged snapshots: 1. Thread A enters ExpireSnapshotsImpl.expireUntil. 2. Thread A executes tagManager.taggedSnapshots in 1.3.1, or collectTaggedSnapshots on master, and is paused. Snapshot S has no tag at this point. 3. Thread B calls table.createTag with tag name tag-s and snapshot id S. The tag file is written successfully. 4. Thread B returns success. 5. Resume Thread A. 6. Thread A continues data-file and manifest cleanup using the old tagged-snapshot list, which does not contain tag-s. 7. Thread A finally deletes the snapshot metadata for S. 8. Read tag-s using tag time travel. The tag file remains present, but its manifest list, manifest, or data files may have been deleted, causing FileNotFoundException or an unreadable tag. The relevant ordering in release 1.3.1 is: - ExpireSnapshotsImpl reads tagManager.taggedSnapshots once. - The same fixed list is used by createDataFileSkipperForTags and findSkippingTags. - cleanUnusedDataFiles and cleanUnusedManifests run. - Snapshot metadata is deleted last. Tag creation independently does: - AbstractFileStoreTable.findSnapshot loads S. - TagManager.createTag writes the tag metadata through FileIO.overwriteFileUtf8. There is no shared lock, GC epoch, revalidation, or second tag collection between these paths. Current master still collects tagged snapshots once before planning and deleting data and manifest files, and deletes snapshot metadata last. ### What does not meet your expectations? A successfully created tag should remain readable and must protect every file referenced by its tagged snapshot. Snapshot expiration currently protects only tags visible when expiration initially collects tag metadata. A tag created after that point can return success while the same expiration operation deletes files referenced by it. This is a TOCTOU race that can cause data loss from the perspective of the new tag. Checking snapshot existence before and after creating the tag is not sufficient because snapshot metadata is deliberately deleted after data and manifest cleanup. Both checks can succeed before the expiration operation removes the snapshot metadata and referenced files. ### Anything else? Related issues and pull requests address adjacent cases but not this race: - #1288 and #1278 protect tags that already exist when expiration starts. - #3650 handles mutable tag state around snapshot expiration and tag deletion, but not concurrent creation of a new tag after the expiration protection set has been collected. - #5229 and #5237 avoid unsafe deletion when reading tag manifests fails. Possible solutions may require serializing createTag with snapshot expiration, introducing an expiration fence or epoch, or revalidating newly created tags before destructive cleanup. The key requirement is to define a linearization relationship between adding a tag reference and deleting files during snapshot expiration. ### Are you willing to submit a PR? Not committed at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
