JingsongLi opened a new pull request, #9197: URL: https://github.com/apache/paimon/pull/9197
## What changed - introduce a typed data-evolution row-range conflict for retry eligibility - retry Spark data-evolution compaction only for that conflict type - rebase compatible concurrent partial-column files onto the staged compact row-id boundaries - validate concurrent files by their portable file contract instead of `Snapshot.operation` ## Why Data-evolution compaction can race with `MERGE INTO`: compaction changes row-id range boundaries while MERGE writes ordinary partial-column files against the previous boundaries. The commit then fails even though the concurrent changes can be preserved by rewriting only those partial columns onto the new compact ranges. Python MERGE does not persist `Snapshot.operation`, so operation-based detection would miss valid conflicts. This change recognizes the safe file shape directly while retaining the typed row-range conflict as the only retry trigger. Other conflict types still fail immediately. ## Impact Spark compaction can recover from compatible concurrent Spark or Python MERGE updates without rerunning the full compaction. The retry reads the latest snapshot, includes the concurrent partial-column files, and rewrites only the affected fields and row-id ranges. ## Validation - targeted `ConflictDetectionTest` row-range conflict tests - Spark 3.5 `CompactProcedureTest`: `rebase data evolution compact after operation-less partial update` - `mvn -pl paimon-spark/paimon-spark-common -Pspark3 -DskipTests compile` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
