zhuxiangyi opened a new pull request, #966:
URL: https://github.com/apache/paimon-rust/pull/966
### Purpose
Linked issue: #964 (part 2 of 4)
> Stacked on #965. Only the last commit, "feat(table): expire snapshots
after commit", is new here.
Java runs snapshot expiration after each commit
(`TableCommitImpl#maintain`), which is what keeps
tables from growing without bound. paimon-rust commits never did. With the
core expiration from
#965, this PR runs expiration after every commit, like Java does.
### Brief change log
- `TableCommit` calls `maintain()` after every successful commit: append,
overwrite, partition
truncate or drop, and table truncate. `maintain()` runs `ExpireSnapshots`
with the table
options. Like Java's `expireForEmptyCommit` default, this does not depend
on whether the commit
created a snapshot.
- Skipped when:
- `write-only` is set (falling back to `write.compaction-skip`), as in
Java;
- the changelog lifecycle is decoupled (`changelog.num-retained.min/max` or
`changelog.time-retained` above the snapshot settings). Java then moves
expired snapshots into
long-lived changelogs, and paimon-rust has no changelog manager yet.
Expiring here would drop
changelogs the table is configured to keep.
- The commit has already succeeded when expiration runs, so an expiration
error is logged and
retried by the next commit instead of failing the commit. Java's batch
mode would surface it.
Returning an error for a commit that did publish its snapshot invites
callers to retry and
duplicate data.
- New `CoreOptions`: `write_only()` and `changelog_lifecycle_decoupled()`.
- With the defaults (keep at least 10 snapshots, and anything younger than 1
h), short-lived
tables and existing tests are unaffected.
### Tests
- `commit_expires_snapshots`: with `snapshot.num-retained.max = 2`, four
commits leave snapshots
3 and 4. The file-set invariant from #965 holds and the data is correct.
- `commit_keeps_recent_snapshots_by_default`: 12 quick commits keep every
snapshot.
- `commit_skips_expiration`: `write-only`, `write.compaction-skip`, and a
decoupled changelog each
keep every snapshot.
- `failed_expiration_does_not_fail_the_commit`: an invalid retention option
does not fail
commits.
- Unit tests for `changelog_lifecycle_decoupled` and `write_only`.
- Two tests from #965 now set `write-only`, because they need commits not to
expire.
- `cargo test -p paimon --all-targets --features fulltext,vortex` passes
(3782 tests), and so does
clippy with `-D warnings`. `paimon-datafusion --all-targets` passes except
for tests that need
the Spark-provisioned tables (`make docker-up`) or the Lumina native
library, which fail the same
way on `main` locally.
### API and Format
No new public API beyond two `CoreOptions` accessors. Behavior change:
commits now expire
snapshots according to the table's retention options, as in Java.
### Documentation
`docs/src/sql.md` (`expire_snapshots` section) describes automatic
expiration and how to turn it
off with `write-only`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
