JingsongLi opened a new pull request, #9096: URL: https://github.com/apache/paimon/pull/9096
## What changed - Read the latest values for affected row-ID ranges before rewriting stale Spark partial-column updates. - Overlay staged values by `_ROW_ID`, preserving rows newly included by concurrent compaction. - Scope affected splits per write-column group and keep the partial-writer column order stable. - Extend the compaction rebase regression test to update only one of two compacted rows. ## Why The Spark conflict rewriter only regrouped staged rows onto the latest `_FIRST_ROW_ID` boundaries. When compaction merged the staged range with additional rows, `DataEvolutionTableDataWrite` filled the missing positions from a staged row, which could overwrite unaffected values. PyPaimon's `TableUpdateByRowId` avoids this by reading current values and filling the complete target range before writing. This change gives the Spark rewrite path the same consistency behavior: current rows are retained unless a staged update with the same `_ROW_ID` replaces them. ## Validation - `mvn -pl paimon-spark/paimon-spark-common -Pfast-build -Pspark3 -DskipTests compile` - `mvn -pl paimon-spark/paimon-spark-common,paimon-spark/paimon-spark-ut -DskipTests spotless:check` - The updated `Data Evolution: rebase staged merge updates after concurrent compact` Spark 3 regression passed locally. A later broader suite run hit an unrelated local JDK 8 CodeCache/ServiceLoader failure after this regression had passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
