XiaoHongbo-Hope opened a new issue, #9352: URL: https://github.com/apache/paimon/issues/9352
### Description Data evolution self-merge validates staged row-ID partial updates using the base snapshot ID, but not the snapshot UUID or lineage. Both the current Spark implementation and the PyPaimon Ray implementation in #9339 treat snapshot IDs as sufficient identity. A rollback can delete newer snapshots, after which new commits reuse the same numeric snapshot IDs. A staged update from the old snapshot can therefore be accepted against a different replacement snapshot with the same ID (an ABA problem). ### Reproduction This has been reproduced with Spark 3.5: 1. Enable row tracking and data evolution. 2. Create snapshot 1 containing `(id=1, b=10)`. 3. Create snapshot 2 containing `(id=2, b=20)`. 4. Stage a self-merge partial update `b = b + 1` for `id=2`, based on the original snapshot 2. 5. Roll back the table to snapshot 1. 6. Insert `(id=30, b=300)`, recreating snapshot ID 2 with a different UUID and reusing the relevant row ID. 7. Commit the old staged update. Expected: `(id=30, b=300)` remains unchanged, or the old commit fails closed. Actual: the commit succeeds and produces `(id=30, b=21)`. The old update is silently applied to the replacement row. ### Trigger window and impact The trigger window is narrow: a self-merge must remain staged across a rollback, snapshot IDs must be recreated, and row-ID ranges must overlap. However, the impact is a silent wrong-row update rather than a clean commit failure. ### Follow-up scope Fix Spark and Python together so that their behavior remains aligned: - fail closed when `latest snapshot ID < base snapshot ID`; - detect equal snapshot IDs with different snapshot UUIDs; - prevent rollback snapshot-ID reuse from accepting staged updates from an old lineage; - decide how the base snapshot UUID is carried through self-merge staging and commit conflict detection; - add equivalent Spark and Python regression tests for `latest < base`, same-ID/different-UUID, and rollback ABA. PR #9339 intentionally does not address this lineage problem. It remains aligned with the current Spark behavior and is scoped to forward snapshot rebase, conflict retry, and safe abort/temporary-file cleanup. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
