JingsongLi opened a new pull request, #1259:
URL: https://github.com/apache/paimon-rust/pull/1259
### Purpose
Rust table commits currently publish data without triggering compaction,
including when callers set `write-only=false`. Run one synchronous compaction
round after successful data publication, with `write-only` defaulting to
`false`. Setting it to `true` leaves compaction to a dedicated job.
### Brief change log
- Parse `write-only` and its legacy `write.compaction-skip` alias, with the
canonical option taking precedence.
- Reuse the existing planner/executor after append, overwrite and
state-dependent data commits. Publish eligible results in a separate `COMPACT`
snapshot; empty plans, truncation and unsupported table modes do not trigger
automatic publication.
- Give automatic compaction a separate commit identity tied to the data
snapshot, so checkpoint recovery and repeated batch commits do not duplicate
data or completed compaction snapshots.
- Resolve the current schema for compaction while retaining explicit dynamic
overrides and the writer's historical read view.
- Preserve committed data and the original error when compaction fails.
Recovery uses the same commit user and identifier.
- Set `write-only=true` explicitly in fixtures that assert raw file layouts,
physical row positions or fixed snapshot sequences.
### Tests
- Added eight behavioral tests covering the default and explicit settings,
the legacy alias, append thresholds, fixed/dynamic PK updates, overwrite,
historical schema evolution, Data Evolution row IDs, unsupported modes and
failure recovery.
- A full test sweep plus focused reruns after fixture updates verified 4,700
unit tests (6 ignored) and all 79 integration targets with Vortex enabled:
`cargo test --locked -p paimon --all-targets --features vortex
--no-fail-fast -- --test-threads=8`.
- Focused reruns of `pk_vector_baseline_test`, `pk_vector_batch_test`,
`rest_catalog_test`, `row_variant_test` and `rowkind_field_test` passed after
adapting their fixtures.
- `cargo fmt --all -- --check` and `git diff --check` passed.
- Strict Clippy encounters four pre-existing `clippy::nonminimal_bool`
findings in unchanged code. All targets pass with that lint alone allowed:
`cargo clippy --locked -p paimon --all-targets --features vortex -- -D
warnings -A clippy::nonminimal_bool`.
### API and Format
Adds `CoreOptions::write_only()` and changes supported table commits to
compact by default. Data publication and compaction remain separate
transactions, so a compaction failure does not roll back committed data. Uses
existing snapshot and file formats; does not add background scheduling or
snapshot expiration.
### Documentation
Updated `docs/src/compaction.md` with defaults, planner behavior, supported
modes, failure semantics and recovery APIs.
Implementation and regression tests were prepared with Codex assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
