JingsongLi opened a new pull request, #1256: URL: https://github.com/apache/paimon-rust/pull/1256
### Purpose Follow-up to #1255 and the remaining first-row slice of #1251. Enable incremental compaction for non-DV `merge-engine=first-row` tables with `changelog-producer=none` or `lookup`. A small L0 write can become visible while large accepted history stays in place. ### Brief change log - Reuse the existing universal run-selection policy and local Lookup File cache for first-row. Both producers now require `local-cache.enabled=true` on the actual catalog/FileIO. - Mark point-lookup results from retained files. Keep accepted keys in those files and omit their duplicate compact output, while preserving selected positive-level rows in the rewritten run. Validate selected input records before applying the retained-history shortcut. - Preserve the first accepted add's kind and sequence for newly accepted lookup changelog rows. Allow compaction to retire L0 inputs without data or changelog outputs when every key already exists in retained history. - Keep the flat task output level and existing captured-snapshot commit validation/recovery. No commit task transport or additional DV protocol is introduced. ### Tests Three integration regressions cover both producers and fixed/dynamic buckets, cross-batch lookup, unchanged retained L5 history and old snapshots, first-add kind/sequence, serialized tasks/results, empty output with omitted task output-level metadata, concurrent L0 appends, idempotent retries, and selection of an intermediate positive run. Cold/warm execution deletes the original history object and uses a new executor to prove generated Lookup Files are consumed. Existing tests now explicitly exercise positive-only first-row maintenance and cache requirements for first-row with both producers. The baseline fails before the incremental selection change. Two semantic mutations also fail as expected: bypassing the lookup-hit omission writes 3 rows instead of 2; treating selected positive rows as retained lookup hits writes 1 instead of 2. Both mutations were restored. 4,674 library unit tests and 196 related integration tests passed with `--test-threads=4`; six existing library tests were ignored. All 73 PK compaction tests passed. `cargo fmt --all -- --check`, `git diff --check`, and `cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool` passed. The allowance covers existing repository findings with the local toolchain. Behavior was checked against Java `LookupStrategy`, `FirstRowMergeFunctionWrapper`, `FirstRowMergeFunction`, and `LookupMergeTreeCompactRewriter` at Apache Paimon commit `5be7e2dd1f`. ### API and Format No new public API, result fields, data format or Lookup File format. First-row compaction now requires actual FileIO local-cache even with `changelog-producer=none`. Newly planned first-row tasks use incremental run selection and carry `output_level`; old executable tasks without it must be replanned. Prepared results with matching file changes and missing task output-level metadata remain supported, including empty results and retries. ### Documentation Updated the compaction guide with first-row incremental selection, local-cache requirements, retained-history behavior and empty outputs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
