JingsongLi opened a new pull request, #10969:
URL: https://github.com/apache/paimon/pull/10969

   ### Purpose
   
   Concurrent Spark MERGE INTO statements on Data Evolution tables fail when 
intervening APPEND commits update the same row-ID ranges and columns. Retry 
deterministic matched updates and upserts against the latest snapshot using the 
existing commit retry limits.
   
   - Recompute affected file groups while retaining unaffected staged update 
files. Account for changes to ON columns, matched conditions, and assignment 
inputs; preserve file reuse when concurrent writers change unrelated columns.
   - For upserts, update rows that became matched after concurrent inserts, 
including rows in new partitions. Recompute NOT MATCHED rows against the 
complete latest target, regenerate insert files, and abort superseded staged 
insert files.
   - Keep the existing restrictions for DELETE, nondeterministic expressions, 
expression subqueries, explicit time travel, deletion vectors, existing-row 
BLOB/VECTOR updates, and update index-file changes. Preserve the separate Spark 
V1 UPDATE retry policy and document the behavior.
   
   This does not add business-key uniqueness: concurrent inserts that do not 
raise a row-ID update conflict remain outside this recovery path.
   
   ### Tests
   
   Local verification, with formatting and static checks enabled:
   
   - Spark 3.5 / JDK 8: 95 tests passed (22 focused tests and 73 RowTracking 
tests).
   - Spark 4.1 / JDK 17: 22 focused tests passed with `-Dslf4j.version=2.0.16`.
   - Spark 4.0 / JDK 17: main and test compilation passed with `-DskipTests 
test`.
   
   The RowTracking run used the packaged `paimon-codegen-loader` plugin. Shared 
Spark outputs were cleaned when switching Scala/JDK versions.
   
   Regressions cover null-safe composite keys, COALESCE assignments, source 
persistence on/off, changed ON keys and expression inputs, new partitions, 
repeated conflicts, exact reuse of unaffected file names, superseded 
insert-file cleanup, retry limits, and overwritten row-ID rejection.
   
   ```sh
   mvn -o -pl paimon-spark/paimon-spark-3.5 -am -Pspark3 \
     -DfailIfNoTests=false -Dtest=none -DCodeCacheSize=512m \
     
-DwildcardSuites=org.apache.paimon.spark.commands.DataEvolutionMergeRetryTest,org.apache.paimon.spark.commands.DataEvolutionUpdateSnapshotTest,org.apache.paimon.spark.commands.MergeIntoPaimonDataEvolutionTableTest,org.apache.paimon.spark.sql.RowTrackingTest
 test
   
   mvn -o -pl paimon-spark/paimon-spark-common,paimon-spark/paimon-spark-ut 
-Pspark4 clean
   
   mvn -o -pl paimon-spark/paimon-spark-ut -am -Pspark4 \
     -Dslf4j.version=2.0.16 -DfailIfNoTests=false -Dtest=none 
-DCodeCacheSize=512m \
     
-DwildcardSuites=org.apache.paimon.spark.commands.DataEvolutionMergeRetryTest,org.apache.paimon.spark.commands.DataEvolutionUpdateSnapshotTest,org.apache.paimon.spark.commands.MergeIntoPaimonDataEvolutionTableTest
 test
   
   mvn -o -pl paimon-spark/paimon-spark-4.0 -am -Pspark4 \
     -Dslf4j.version=2.0.16 -DskipTests test
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to