JingsongLi opened a new pull request, #1249: URL: https://github.com/apache/paimon-rust/pull/1249
### Purpose Extend the existing Rust compaction API to Data Evolution append tables, which `CompactBuilder` currently rejects. Merge normal-column files while preserving stable row IDs, sequence metadata and logical deletions. Follow-up to #1247; related to #286 as a Rust maintenance prerequisite. Python bindings remain separate. ### Brief change log - Plan complete aligned row-ID groups and pack adjacent ranges using Java's Data Evolution binning rules. Support `compaction.min.file-num=1` for single-file rewrites while ordinary append still requires at least two files. - Stream merged normal-column values into one output per task, retaining input sequence bounds and per-column sequence metadata. Preserve dedicated Blob/Vector files, inline descriptors/views and global indexes. - Move deletion vectors only from touched index files, rebasing on the latest snapshot on each commit attempt. Reject concurrent normal-column changes while allowing disjoint appends and preserving concurrent deletions. - Preserve serialized task subsets and idempotent recovery, including CAS loss and lost publication responses. - Reject rewrites that would discard historical inline values after a dedicated-storage option change, or exceed the current Rust deletion-vector writer's addressable local row positions. ### Tests - 64 compaction integration tests passed: 13 Data Evolution, 14 append and 37 primary-key cases. - Core library regression: 4,639 passed, 6 ignored; focused compaction/range unit tests and the added CAS/publication-response recovery test also passed. - Java reference tests: 65 cases passed across `DataEvolutionCompactCoordinatorTest` and `DataEvolutionNormalCompactTaskTest`. - `cargo fmt --all --check` and `git diff --check` passed. - Focused Clippy passed with `-D warnings -A clippy::nonminimal_bool`. Strict Clippy reports four existing `nonminimal_bool` diagnostics in unchanged files. Coverage includes four data formats, nulls, historical snapshots, field renames/name reuse/type promotion, nested updates, dedicated files, index freshness, both DV storage layouts, independent subsets and concurrent commits. A Java/Rust output-artifact roundtrip was not run. ### API and Format Reuse `new_compact_builder()` and `TableCommit::commit_compact`; no new public API or storage-format version. Scope is normal files in unaware Data Evolution append tables with row tracking and no primary key. Dedicated Blob/Vector compaction, large-file splitting and deletion materialization remain unsupported. With deletion vectors enabled, a compact output may contain at most 4,294,967,296 physical row slots until 64-bit DV writing is available. ### Documentation Update `docs/src/compaction.md` with Data Evolution planning, row/sequence preservation, deletion-vector handling, concurrency and supported limits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
