JingsongLi opened a new pull request, #8915: URL: https://github.com/apache/paimon/pull/8915
## What changed - detect row-ID existence conflicts caused by concurrent compaction - automatically rebase regular partial-column update files onto current file boundaries - build rewrite file indexes directly from the checked snapshot entries - bound automatic rewriting by a configurable size limit, defaulting to 256 MiB - keep rewrite retries within the existing commit timeout and retry budget - document recovery behavior and unsupported deletion-vector or dedicated-file cases ## Why Partial-column updates retain the row-ID range of the files they read. If compaction merges or splits those files before commit, the staged update references obsolete file boundaries and fails with a row-ID existence conflict even though the logical rows still exist. Rewriting the staged update against the checked current entries allows the commit to proceed without hiding real concurrent updates. ## User impact PyPaimon now recovers automatically from eligible concurrent compaction conflicts when the affected current files are within `data-evolution.row-id-conflict-rewrite.max-size`. The default is `256 MB`; setting it to `0 B` disables recovery. Logical update conflicts, removed row IDs, deletion-vector tables, and existing-row BLOB/VECTOR updates continue to use the normal conflict path. ## Validation - `file_store_commit_test.py`: 10 passed - `table_upsert_by_key_test.py`: 67 passed, 6 subtests passed - `table_update_test.py` and `conflict_detection_test.py`: 128 passed, 17 subtests passed - `shard_table_updator_test.py`: 8 passed - Java reactor/Core E2E: 1 passed - Java compaction followed by Python rewrite commit E2E: 1 passed - Python syntax and diff checks passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
