JingsongLi opened a new pull request, #11287:
URL: https://github.com/apache/paimon/pull/11287

   ### Purpose
   
   A Spark Data Evolution centroid build can train on snapshot 1 and publish 
after a concurrent `CALL sys.rewrite_file_index`. The file-index rewrite emits 
a DELETE and an ADD for the same physical data file. The source check 
introduced in #11249 interprets both entries as vector data changes and rejects 
the build.
   
   Match replacement pairs within each COMPACT delta by partition, bucket and 
file name. Skip a unique pair only when the total bucket count matches and the 
full data metadata compares equal after removing embedded and external file 
indexes. Continue checking the other entries in the same snapshot.
   
   ### Tests
   
   - Added a commit regression covering an embedded-to-external file-index 
rewrite, row-ID reassignment, and mixed compaction entries.
   - Ran `ConflictDetectionTest` and `FileStoreCommitTest` on Java 8, plus 
Spotless and Checkstyle.
   - Replayed a Spark SQL race with the centroid writer paused before 
publication. Rewriting the scalar Bloom filter kept the original data path and 
row values. The same build that previously failed now published successfully.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to