JingsongLi opened a new pull request, #1258: URL: https://github.com/apache/paimon-rust/pull/1258
### Purpose Compaction currently rejects primary-key tables with `bucket=-2`, leaving postpone files and fixed-bucket Level-0 data pending. Support publishing this data through the existing `CompactBuilder` API, including deletion-vector-enabled `deduplicate` tables with `none`, `input`, or `lookup` changelog producers. Linked issue: none. ### Brief change log - Plan snapshot-bound partition rewrites, preserve existing bucket counts or infer counts for new partitions, and route pending records into real buckets. - Preserve writer order and row kinds, sort bounded batches with multi-pass temporary-run merging, and reuse PK merge and lookup changelog generation. - Atomically publish inputs and outputs with partition subsets, serialized results, idempotent retries, and conflict checks for concurrent real-bucket writes. - Apply captured deletion vectors before routing and merging; retire rewritten DV ranges while preserving unrelated partitions and concurrent pending masks. - Validate ROW record boundaries so both Rust pending files and Java KV physical layouts can be decoded safely. Postpone DV compaction retains the existing requirement for catalog/FileIO local-cache. It supports `deduplicate`; other merge engines and global index manifests remain unsupported in DV mode. Existing partitions are not rescaled. ### Tests - `cargo test --locked -p paimon --test pk_compact_test --test append_compact_test --test data_evolution_compact_test --test data_evolution_compact_layout_test --test postpone_native_write_test --test dv_merge_engine_write_test`: 140 passed. - `cargo test --locked -p paimon --lib arrow::format::row::tests`: 28 passed; 1 manual benchmark ignored. - `cargo fmt --all --check` and `git diff --check`: passed. - `CARGO_INCREMENTAL=0 cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool`: passed. Existing repository `nonminimal_bool` warnings are allowed; other warnings are denied. Regression coverage includes all four file formats, merge engines, lookup before/after images, writer timestamp ties, 600 temporary sort runs, Java ROW layouts, pending/L0/history deletion masks, shared DV containers, empty outputs, concurrent conflicts, and subset/retry behavior. ### API and Format Uses the existing public compaction API and persistent data/index formats. Serialized tasks/results gain defaulted source/output bucket mappings for partition rewrites. DV results retain explicit deletion metadata for first publication. ROW decoding now rejects layouts that do not consume exactly one physical record. ### Documentation Updated `docs/src/compaction.md` with postpone planning, bucket-count precedence, execution, concurrency, deletion-vector behavior, and current limits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
