JingsongLi opened a new pull request, #11001: URL: https://github.com/apache/paimon/pull/11001
### Purpose Concurrent automatic compactions on primary-key tables with deletion vectors can both pass file-level checks when they rewrite different inputs from the same stale partition state. This can make duplicate primary keys visible. For `deletion-vectors.enabled = true` and `write-only = false`, carry the writer's restored snapshot in the existing commit message field. Before each COMPACT publication attempt, reject a newer COMPACT or OVERWRITE from another commit user that changed the same partition. This also detects changes in different buckets of that partition and allows different partitions to progress independently. APPEND and publication retries preserve the writer's baseline. The check reads existing data and index manifests and caches verified intervals in the committer. Flink precommit compaction and the Flink and Spark postpone paths preserve the baseline through message rewrites. Changelog-only and file-index-only rewrites can commit without a compaction baseline. The concurrency guarantee applies to ordinary automatic compaction with fixed bucket routing, distinct commit users, and available snapshot and manifest history. ### Tests - Core regression suites passed 142 cases, including same-partition conflicts, cross-partition isolation, APPEND and retry baselines, and DV removal. - Flink passed 7 focused operator and postpone integration cases and 3 writer recovery cases. - Spark 3 passed all 30 cases in `PostponeBucketTableTest`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
