JingsongLi opened a new pull request, #10969:
URL: https://github.com/apache/paimon/pull/10969
### Purpose
Concurrent Spark MERGE INTO statements on Data Evolution tables fail when
intervening APPEND commits update the same row-ID ranges and columns. Retry
deterministic matched updates and upserts against the latest snapshot using the
existing commit retry limits.
- Recompute affected file groups while retaining unaffected staged update
files. Account for changes to ON columns, matched conditions, and assignment
inputs; preserve file reuse when concurrent writers change unrelated columns.
- For upserts, update rows that became matched after concurrent inserts,
including rows in new partitions. Recompute NOT MATCHED rows against the
complete latest target, regenerate insert files, and abort superseded staged
insert files.
- Keep the existing restrictions for DELETE, nondeterministic expressions,
expression subqueries, explicit time travel, deletion vectors, existing-row
BLOB/VECTOR updates, and update index-file changes. Preserve the separate Spark
V1 UPDATE retry policy and document the behavior.
This does not add business-key uniqueness: concurrent inserts that do not
raise a row-ID update conflict remain outside this recovery path.
### Tests
Local verification, with formatting and static checks enabled:
- Spark 3.5 / JDK 8: 95 tests passed (22 focused tests and 73 RowTracking
tests).
- Spark 4.1 / JDK 17: 22 focused tests passed with `-Dslf4j.version=2.0.16`.
- Spark 4.0 / JDK 17: main and test compilation passed with `-DskipTests
test`.
The RowTracking run used the packaged `paimon-codegen-loader` plugin. Shared
Spark outputs were cleaned when switching Scala/JDK versions.
Regressions cover null-safe composite keys, COALESCE assignments, source
persistence on/off, changed ON keys and expression inputs, new partitions,
repeated conflicts, exact reuse of unaffected file names, superseded
insert-file cleanup, retry limits, and overwritten row-ID rejection.
```sh
mvn -o -pl paimon-spark/paimon-spark-3.5 -am -Pspark3 \
-DfailIfNoTests=false -Dtest=none -DCodeCacheSize=512m \
-DwildcardSuites=org.apache.paimon.spark.commands.DataEvolutionMergeRetryTest,org.apache.paimon.spark.commands.DataEvolutionUpdateSnapshotTest,org.apache.paimon.spark.commands.MergeIntoPaimonDataEvolutionTableTest,org.apache.paimon.spark.sql.RowTrackingTest
test
mvn -o -pl paimon-spark/paimon-spark-common,paimon-spark/paimon-spark-ut
-Pspark4 clean
mvn -o -pl paimon-spark/paimon-spark-ut -am -Pspark4 \
-Dslf4j.version=2.0.16 -DfailIfNoTests=false -Dtest=none
-DCodeCacheSize=512m \
-DwildcardSuites=org.apache.paimon.spark.commands.DataEvolutionMergeRetryTest,org.apache.paimon.spark.commands.DataEvolutionUpdateSnapshotTest,org.apache.paimon.spark.commands.MergeIntoPaimonDataEvolutionTableTest
test
mvn -o -pl paimon-spark/paimon-spark-4.0 -am -Pspark4 \
-Dslf4j.version=2.0.16 -DskipTests test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]