JingsongLi opened a new pull request, #1253: URL: https://github.com/apache/paimon-rust/pull/1253
### Purpose Follow-up to #1252, extracted from #1251. Small writes to a deduplicate primary-key table currently cause compaction to rewrite the captured bucket, including large historical files. Incremental sorted-run selection now leaves large higher-level runs in place and rewrites only the selected runs. This PR enables the policy for `merge-engine=deduplicate` with `changelog-producer=none/input` and no deletion vectors. Other modes retain their merged behavior while their lookup/cache dependencies are split separately. ### Brief change log - Port Java UniversalCompaction run grouping, forced L0 progress, size amplification, size ratio, run-count maintenance and output-level selection. Match Java's L0 ordering, including sequence and creation-time tie-breakers, and accept a compaction trigger of one. - Capture the selected output level in serialized tasks. Recompute the selected runs against the captured snapshot before execution and publication to reject altered selections. - Preserve winning retracts and user sequences across partial-run rewrites, retain concurrent L0 appends, and keep completed legacy results recoverable by idempotent retry. Uncommitted legacy tasks without selection metadata must be replanned. - Add focused behavior and boundary regressions, including an order-sensitive aggregation example that must retain its history in the merge. ### Tests 4,661 library unit tests and 159 related integration tests passed; six existing unit tests were ignored. The integration run covers all 60 PK compaction tests plus append/Data Evolution compaction, DV writes, incremental/streaming reads and partial-update state. Six new planner tests cover stable run grouping/order, Java threshold boundaries, singleton/empty inputs, existing high levels, large-byte arithmetic and configuration bounds. Six new integration tests cover untouched history files, data and system-sequence equivalence, historical snapshots, user-sequence winners, tombstones, tampered tasks/results, concurrent L0 appends, retry recovery and the aggregation scope boundary. `cargo fmt --all -- --check`, `git diff --check`, and `cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool` passed. The Clippy allowance covers existing repository findings with the local toolchain. Selection was checked against Java `UniversalCompaction`, `Levels` and `CoreOptions` at Apache Paimon commit `5be7e2dd1f`. ### API and Format Adds defaulted internal `primary_key.output_level` metadata to serialized compaction tasks. Table data formats are unchanged. No full-compaction option is added. ### Documentation Updated the compaction guide with the enabled modes, run-selection options, output levels and serialized-task recovery behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
