JingsongLi opened a new pull request, #1247: URL: https://github.com/apache/paimon-rust/pull/1247
### Purpose Add a Rust API for consolidating table files and atomically publishing the replacement files in a COMPACT snapshot. Planning, execution and commit are separate so applications can schedule tasks across workers and commit completed subsets. Related to #286: this provides the Rust compaction prerequisite; Python maintenance bindings remain separate. ### Brief change log - Add `CompactBuilder`, `CompactPlanner`, `CompactExecutor`, serializable plans/tasks/results, and `TableCommit::commit_compact`. - Merge small files within append-table partitions/buckets. Rewrite complete PK buckets with deduplicate, first-row, partial-update and aggregation, preserving merge state, sequence numbers and tombstones. - Support fixed and dynamic PK buckets, including cross-partition dynamic buckets. Preserve HASH assignments and carry forward the latest HASH manifest when writes commit concurrently. - Validate captured inputs, schema and file metadata; reject conflicting rewrites; make retries of the same task subset idempotent. Rebuild file indexes and clean up unsubmitted outputs on execution failure. ### Tests Passed 68 relevant integration tests, including 10 append compaction cases, 27 PK compaction cases (7 dynamic-bucket cases), and the existing dynamic, cross-partition and precomputed-bucket write suites: ```text cargo test --locked -p paimon --test append_compact_test --test pk_compact_test --test dynamic_bucket_write_test --test cross_partition_write_test --test precomputed_bucket_write_test ``` Also passed 142 table-commit unit tests and formatting/lint checks: ```text cargo test --locked -p paimon --lib table::table_commit::tests cargo +1.98.0 fmt --all -- --check cargo +1.98.0 clippy --locked -p paimon --lib --test pk_compact_test --test append_compact_test -- -D warnings ``` Coverage includes historical reads, incremental events, four data file formats, merge state across subsequent writes, concurrent commits, partition filtering, HASH assignment history, writer restarts, cross-partition migrations, index rebuilding, memory limits and failure cleanup. ### API and Format Adds the public Rust compaction API. Uses existing data-file, manifest and COMPACT snapshot formats. Scope is ordinary append tables and PK tables with `changelog-producer=none/input` (first-row requires `none`). Row tracking, data evolution, deletion vectors, query authorization, rowkind fields, postpone buckets, dedicated blob/vector storage, non-HASH snapshot indexes, and `lookup/full-compaction` changelog generation remain unsupported. PK compaction uses the existing 256 overlapping merge-input limit. ### Documentation Add `docs/src/compaction.md` and its navigation entry, covering the API workflow, concurrency/retry behavior, merge-engine semantics, dynamic buckets and current limits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
