zhoulii opened a new issue, #9596: URL: https://github.com/apache/paimon/issues/9596
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 2.0 ### Compute Engine Spark ### Minimal reproduce step 1. Enable Spark V1 writes and create a data evolution table with row tracking. 2. Insert `(1, 'pending')`. 3. Start `UPDATE t SET status = 'done' WHERE status = 'pending'` and let it pin snapshot N. 4. Concurrently execute `UPDATE t SET status = 'cancelled' WHERE id = 1`, producing snapshot N+1. 5. Resume the first UPDATE. ### What doesn't meet your expectations? The first UPDATE may read its source from snapshot N but read its target from snapshot N+1. It then uses snapshot N+1 as the conflict-detection baseline, so the concurrent change in N+1 is not checked and may be overwritten. The final status can become `done`, which cannot be produced by either serial execution order. The concurrent update should instead be detected as a conflict. ### Anything else? [#8411](https://github.com/apache/paimon/pull/8411) introduced snapshot pinning for V1 UPDATE on data evolution tables so that row selection and conflict detection are based on a stable snapshot. However, the current data evolution MERGE path does not preserve this guarantee end to end. Although the UPDATE source is pinned to snapshot N, the target scan independently resolves the latest snapshot and also uses it as the conflict-detection baseline. If a concurrent write produces snapshot N+1 in between, the command may use: ```text source snapshot = N target snapshot = N+1 conflict-detection baseline = N+1 ``` Because conflict detection starts after its baseline, the concurrent change contained in N+1 is not checked. This weakens the snapshot consistency guarantee established by #8411 and may produce a result that cannot be explained by any serial execution order. The proposed fix restores the required invariant: ```text source snapshot == target snapshot == conflict-detection baseline ``` It makes the target scan respect the pinned snapshot by resolving it through `TimeTravelUtil.tryTravelOrLatest(table)`. There is also a related execution-plan issue. The self-merge shortcut deliberately falls back for time-travel sources to avoid changing the semantics of user-authored MERGE statements. V1 UPDATE uses the same time-travel option internally to pin its snapshot, so its generated self-merge is also conservatively routed through the general MERGE path, introducing an unnecessary join, shuffle, and sort. The PR distinguishes the safe pinned case by requiring source and target to share the exact same `SparkTable` instance. This preserves the self-merge shortcut for pinned V1 UPDATE without relaxing the fallback behavior for user-authored time-travel MERGE statements. The tests cover concurrent conflict detection, snapshot-consistent UPDATE behavior, the optimized unconditional UPDATE plan, and the existing time-travel MERGE fallback. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
