JingsongLi opened a new pull request, #1252: URL: https://github.com/apache/paimon-rust/pull/1252
### Purpose Extract non-DV lookup changelog production from #1251 into an independently mergeable change (13 files, +1179/-70). This uses the existing primary-key compaction planner and provides the changelog foundation for subsequent deletion-vector and incremental-lookup work. L0 inserts, updates and deletes now produce logical before/after events atomically with compacted data. Incremental and streaming scans can consume the resulting COMPACT snapshot's changelog manifest. ### Brief change log - Add lookup merge wrapping and source-level metadata so the lowest positive level supplies the latest materialized history, avoiding repeated aggregation of older states. - Generate INSERT, UPDATE_BEFORE, UPDATE_AFTER and DELETE events, including first-row insert-only behavior and the Java-compatible changelog filtering/deduplication options. - Close both data and changelog writers before handing off prepared files, clean both sides on failure, serialize changelog results/metrics, and include logs in idempotent commit recovery. - Reject concurrent positive-level history changes, including additions at another level, while allowing concurrent L0 appends. - Require a real enabled catalog/FileIO local-cache for lookup compact executors. ### Tests 372 related tests passed: 146 integration tests covering PK/append/Data Evolution compaction, incremental and streaming reads, DV writes and partial-update state, plus 226 sort-merge, KV-reader and table-commit unit tests. Lookup-specific coverage includes fixed/dynamic buckets, event images, row deduplication/filter flags, first-row, aggregation, materialized-history selection, subset/retry behavior, concurrent history changes, late-read failure cleanup and cache validation. `cargo fmt --all -- --check`, `git diff --check`, and `cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool` passed. The Clippy allowance covers existing repository findings with the local toolchain. ### API and Format Adds `CompactResult::changelog_files()` and changelog counters to `CompactMetrics`; serialized results include a defaulted changelog-file list. Lookup compact executors require `local-cache.enabled=true` on the catalog/FileIO; a table-property flag alone cannot attach a cache. Table data formats are unchanged. ### Documentation Updated the compaction guide and SQL documentation for non-DV lookup changelog production. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
