henriquejsza opened a new pull request, #10897:
URL: https://github.com/apache/paimon/pull/10897
### Purpose
Closes #10883.
All branches of a table share the same physical data, manifest, index and
statistics files; only snapshot, tag and schema metadata are branch scoped
(`FileStorePathFactory` roots data and manifest paths at the table root,
while
`SnapshotManager`/`TagManager` prepend the branch path). Snapshot expiration
on
a branch built its skipping set only from that branch's own snapshots and
tags
(`ExpireSnapshotsImpl#collectTaggedSnapshots` and `manifestSkippingSet`), so
when a branch created from a tag drops its own copy of that tag and then
expires, it deletes files that the main branch still references. After that,
reading main fails with `FileNotFoundException` on a manifest list or data
file, exactly as reported in the issue.
This change makes expiration aware of the other branches. Before deleting
files, `AbstractFileStoreTable#newExpireSnapshots` collects the data files
and
the manifest/manifest-list/index/statistics files referenced by every other
branch's live snapshots and tags, and `ExpireSnapshotsImpl` skips any file
that
is still referenced. The data-file lookup is restricted to the current
expiration batch so memory stays proportional to the batch, mirroring the
existing per-tag skipper. Reading another branch is best effort: if it fails,
the batch is kept rather than risking deletion of in-use files. When the
table
has no other branch, the collector is `null` and the common path is
unchanged.
### Tests
- Added
`AppendOnlySimpleTableTest#testExpireOnBranchKeepsFilesReferencedByMain`,
which reproduces the issue: tag main's snapshot, create a branch from the
tag,
write on the branch, delete the branch's copy of the tag, expire the
branch to
one snapshot, then assert main is still fully readable. It fails on master
(`FileNotFoundException` on main's manifest list) and passes with this
change.
- `mvn -pl paimon-core
-Dtest='AppendOnlySimpleTableTest,ExpireSnapshotsTest,FileDeletionTest' test`
passes.
- `mvn -pl paimon-core spotless:check checkstyle:check` passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
