JingsongLi opened a new pull request, #9091: URL: https://github.com/apache/paimon/pull/9091
## What changed - Rebase stale regular partial-column files produced by Spark Data Evolution `MERGE INTO` onto the latest row-id file boundaries after concurrent compaction. - Commit the rewritten update messages together with the original insert and delete messages without rerunning the MERGE source or join. - Keep logical concurrent-update conflict detection and support both Spark 3 and Spark 4 command paths. - Document the recovery behavior and its exclusions. ## Why Concurrent compaction can rewrite row-id file boundaries after MERGE has staged its partial-column files. The staged files then fail row-id existence validation, causing the whole MERGE command to be retried even though only the staged update files need rebasing. This follows the recovery approach introduced for PyPaimon in #8915, but performs the rewrite as distributed Spark DataFrame processing. Therefore the PyPaimon-only `data-evolution.row-id-conflict-rewrite.max-size` limit does not apply to Spark. Recovery remains disabled for deletion-vector tables and existing-row BLOB or VECTOR staged files. Non-compaction logical conflicts are still reported. ## Validation - Spark 3.5 `RowTrackingTest`: 43 tests passed, including the new concurrent-compaction regression with `commit.max-retries=0` and the PyPaimon size option set to `0 B`. - Spark 4 common and command modules compiled successfully. - Spotless and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
