JingsongLi opened a new pull request, #1251: URL: https://github.com/apache/paimon-rust/pull/1251
### Purpose Support deletion-vector compaction and lookup changelog production for primary-key tables without repeatedly rewriting large historical runs. A small L0 update now looks up its previous state through `local-cache`, writes the merged value and changelog, and marks the previous physical row deleted while preserving the higher-level data file. ### Brief change log - Replace whole-bucket selection with Java-style universal incremental run selection and capture the untouched lookup dependencies. - Generate INSERT, UPDATE_BEFORE, UPDATE_AFTER and DELETE changelogs with the existing merge engines and Java-compatible changelog options; first-row emits inserts for new keys. - Store generated SST indexes and Arrow IPC value blocks in the existing local-cache backend, sharing its namespace, capacity, CRC validation and LRU across memory, disk and shared disk backends. Rebuild evicted or corrupt artifacts and publish descriptors only after completing their blocks. - Atomically merge new historical-position DVs, retire rewritten-file vectors, preserve unrelated indexes, and validate lookup dependencies on commit retries. Keep concurrent L0 appends, historical snapshots and idempotent subset commits working. ### Tests 268 related tests passed: - 166 integration tests covering PK compaction, append and Data Evolution compaction, DV merge engines, incremental/streaming changelogs, and partial-update sequence/delete state. - 102 unit tests covering local-cache and incremental PK planning/lookup, including shared capacity, concurrent misses, cancellation, corruption, reopening and resource limits. - `cargo fmt --all -- --check` and `git diff --check` passed. - `cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool` passed; the allowance covers the repository's existing lint findings with the local toolchain. ### API and Format DV, lookup-changelog and first-row compact executors require an actual cache attached to the catalog/FileIO (`local-cache.enabled=true`). `local-cache.dir` selects disk storage and `local-cache.max-size` governs the shared quota; `lookup.cache-max-disk-size` is rejected. Task/result serialization now carries incremental selection, lookup DV notifications and changelog files. Public APIs expose changelog files and compaction/cache metrics. Table data and DV formats remain unchanged; local Lookup File artifacts use a disposable Rust-private format. Compaction uses incremental run selection exclusively. ### Documentation Updated the compaction guide, local-cache configuration guide and SQL feature documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
