zhuxiangyi opened a new issue, #964:
URL: https://github.com/apache/paimon-rust/issues/964
### Search before asking
- [x] I searched in the issues and found nothing similar.
### Motivation
Java Paimon expires snapshots after every commit and provides
`expire_snapshots` and
`remove_orphan_files` procedures. paimon-rust has none of these, so a table
written through the
Rust API, pypaimon native, or the C FFI keeps every snapshot forever. Each
commit adds a snapshot,
manifest lists, and manifests, and files replaced by overwrites or row-level
rewrites are never
removed. Files left behind by failed or interrupted writes are never cleaned
up either.
### Solution
Port Java's table maintenance in small, independently reviewable PRs, each
following the Java
implementation and keeping its safety rules. (Every read failure deletes
less, never more. A file
still referenced by any snapshot, tag, branch, or consumer is never deleted.)
- [ ] **Snapshot expiration core + `CALL sys.expire_snapshots`**: #PR_A
`ExpireSnapshotsImpl` / `SnapshotDeletion`, with the
`snapshot.num-retained.*`,
`snapshot.time-retained`, and `snapshot.expire.limit` options, consumer
and tag protection, and
Java's procedure arguments.
- [ ] **Expire snapshots after commit**: #PR_B
Like Java's `TableCommitImpl`, run expiration after a commit that creates
a snapshot. It honors
`write-only` and skips when the changelog lifecycle is decoupled
(`changelog.*` retention
longer than snapshot retention), which needs a changelog manager first.
- [ ] **Python API: `Table.expire_snapshots`**: #PR_C
- [ ] **Orphan file cleanup + `CALL sys.remove_orphan_files`**: #PR_D
`LocalOrphanFilesClean`: delete files older than `older_than` (default one
day ago) that no
snapshot, tag, long-lived changelog, or branch references, with a
`dry_run` mode.
Later, not in these PRs:
- a changelog manager and decoupled changelog expiration
(`changelog.num-retained.*`,
`changelog.time-retained`);
- `snapshot.clean-empty-directories` and empty-directory cleanup;
- partition expiration.
### Anything else?
The PRs are stacked: #PR_B, #PR_C and #PR_D each build on #PR_A and are
independent of each other.
### Willingness to contribute
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
