JingsongLi opened a new pull request, #1255: URL: https://github.com/apache/paimon-rust/pull/1255
### Purpose Follow-up to #1254, extracted from #1251. Enable incremental primary-key compaction for `deletion-vectors.enabled=true`, `merge-engine=deduplicate`, and changelog producers `none`, `input` and `lookup`. Small L0 writes can become visible to ordinary DV scans while large historical runs remain in place. ### Brief change log - Extend the existing local Lookup File cache with absolute physical row positions. Cold loading reuses KV schema evolution without merging or filtering physical rows; immutable cached bytes stay independent of snapshot DVs. Apply captured masks to selected inputs and lookup hits, then record physical deletions when historical state moves into the output. - Reuse incremental run selection and the actual catalog/FileIO local-cache for all three producers. Calculate lookup changelogs before dropping retracts from DV data output; DELETE-only compaction can emit no data file. Bound data-file rows to the existing Rust bitmap position range. - Publish data changes, historical DV additions and retired input ranges atomically. Validate selected masks and actual lookup-hit targets, union with the latest bitmaps on each CAS attempt, preserve unrelated ranges sharing a container, and retain HASH/other-bucket indexes. Read each index manifest once and decode/write one bucket at a time. - Keep commit validation based on captured snapshots and file changes. Prepared results carry physical deletion metadata; missing metadata fails first publication while already-committed results can still recover idempotently. Changelog output levels are validated from captured selection even when task output-level metadata and data outputs are absent. - Reject existing files whose persisted schemas were not DV-enabled, including temporary option overrides. No full-compaction API or additional merge engines are enabled. ### Tests Five integration tests cover `none/input/lookup` across fixed and dynamic buckets, exact cross-batch physical DV positions, default scans without merge-on-read, retained history and old snapshots, DELETE-only output and before-image metadata, selected-input mask filtering/retirement, shared index ranges, latest HASH preservation, concurrent L0 writes, independent DV union, changed input/hit masks, removed targets, schema-mode overrides, serialized-result validation and retries. 4,672 library unit tests and 169 related integration tests passed with `--test-threads=4`; six existing unit tests were ignored. All 69 PK compaction tests passed alongside append/Data Evolution compaction, DV writes, incremental/streaming reads and partial-update state. A dense-vector unit regression verifies memory accounting follows bitmap container storage rather than four bytes for every deleted row. `cargo fmt --all -- --check`, `git diff --check`, and `cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool` passed. The allowance covers existing repository findings with the local toolchain. Behavior was checked against Java `LookupLevels`, `LookupMergeTreeCompactRewriter`, `LookupChangelogMergeFunctionWrapper`, `MergeTreeCompactManager` and conflict detection at Apache Paimon commit `5be7e2dd1f`. ### API and Format The compaction API is unchanged. `CompactResult` adds defaulted `deletion_vector_updates`; uncommitted DV results must retain this metadata, including an explicit empty list. `CompactTask` remains flat and commit resolution does not receive tasks. Lookup artifacts advance to private disposable format v2 for physical positions; table data and DV index formats use existing readers/writers. ### Documentation Updated compaction and SQL guides with DV scope, local-cache requirements, physical deletion/commit behavior, row limits and mode-migration restrictions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
