JingsongLi opened a new pull request, #1254: URL: https://github.com/apache/paimon-rust/pull/1254
### Purpose Follow-up to #1253, extracted from #1251. Deduplicate lookup compaction currently selects every active file in a bucket. Use incremental sorted-run selection and fetch retained higher-level history through local Lookup Files, so small L0 updates can generate complete changelogs without rewriting the large history run. ### Brief change log Lookup Files store complete KV rows in disposable SST/IPC artifacts and reuse the actual catalog/FileIO local-cache backend, namespace, eviction and capacity. Memory, disk and shared-disk caches are supported; `local-cache.dir` selects disk storage. Cold reads use the existing schema-evolution decoder, and later executors reuse the generated files. Missing or corrupt blocks rebuild instead of becoming false key misses. Working memory is bounded per executor. Commit validation compares the captured positive-level history with the latest view, including dependencies from unsuccessful lookups, while allowing concurrent L0 appends. Prepared-result recovery continues to validate actual file changes when optional task output-level metadata is absent. This enables incremental lookup for deduplicate without deletion vectors; other merge-engine selection policies remain unchanged. ### Tests Validation covers memory/disk/shared cache reuse and concurrent builds, eviction/corruption and failure recovery, shared download-lock collisions, update/delete/insert payloads and sequence metadata, schema rename/type promotion, history conflicts for hits and misses, and idempotent commits. Integration tests remove the original history object between cold and warm execution to prove the Lookup File is consumed with ordinary data caching disabled. 4,669 library unit tests and 163 related integration tests passed; six existing unit tests were ignored. The integration run covers all 63 PK compaction tests plus append/Data Evolution compaction, DV writes, incremental/streaming reads and partial-update state. `cargo fmt --all -- --check`, `git diff --check`, and `cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool` passed. The Clippy allowance covers existing repository findings with the local toolchain. Behavior was checked against Java `LookupLevels`, `LookupUtils`, `LookupMergeFunction` and `LookupChangelogMergeFunctionWrapper` at Apache Paimon commit `5be7e2dd1f`. ### API and Format Public compaction APIs and table data formats are unchanged. Generated Lookup Files are private, disposable local-cache artifacts. Deduplicate lookup caching accepts `lookup.cache-max-memory-size` and `lookup.cache-spill-compression`; its storage uses `local-cache.max-size`, and `lookup.cache-max-disk-size` is rejected. No full-compaction API is added. ### Documentation Updated the compaction guide with incremental lookup scope, cache backends and budgets, rebuild behavior and commit conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
