JingsongLi opened a new pull request, #11001:
URL: https://github.com/apache/paimon/pull/11001

   ### Purpose
   
   Concurrent automatic compactions on primary-key tables with deletion vectors 
can both pass file-level checks when they rewrite different inputs from the 
same stale partition state. This can make duplicate primary keys visible.
   
   For `deletion-vectors.enabled = true` and `write-only = false`, carry the 
writer's restored snapshot in the existing commit message field. Before each 
COMPACT publication attempt, reject a newer COMPACT or OVERWRITE from another 
commit user that changed the same partition. This also detects changes in 
different buckets of that partition and allows different partitions to progress 
independently. APPEND and publication retries preserve the writer's baseline.
   
   The check reads existing data and index manifests and caches verified 
intervals in the committer. Flink precommit compaction and the Flink and Spark 
postpone paths preserve the baseline through message rewrites. Changelog-only 
and file-index-only rewrites can commit without a compaction baseline.
   
   The concurrency guarantee applies to ordinary automatic compaction with 
fixed bucket routing, distinct commit users, and available snapshot and 
manifest history.
   
   ### Tests
   
   - Core regression suites passed 142 cases, including same-partition 
conflicts, cross-partition isolation, APPEND and retry baselines, and DV 
removal.
   - Flink passed 7 focused operator and postpone integration cases and 3 
writer recovery cases.
   - Spark 3 passed all 30 cases in `PostponeBucketTableTest`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to