JingsongLi commented on code in PR #11048:
URL: https://github.com/apache/paimon/pull/11048#discussion_r4521057496


##########
paimon-python/pypaimon/execution/python/write/commit/row_id_conflict.py:
##########
@@ -35,10 +35,9 @@ def check_row_id_conflicts(table, messages, 
latest_snapshot_id, check_from_snaps
     baseline = manager.get_snapshot_by_id(check_from_snapshot)
     if latest is None or baseline is None:
         raise ValueError('Row-ID conflict check snapshots must exist in the 
current table')
-    current_schema = table.schema_manager.latest()
-    if (baseline.schema_id != latest.schema_id
-            or current_schema is not None and current_schema.id != 
table.table_schema.id):
-        raise ValueError('Row-ID conflict checks require an unchanged, current 
table schema')
+    # Logical conflicts compare field IDs resolved in each file's historical
+    # schema, just like ordinary commits. Different schema versions alone do

Review Comment:
   **[P1] Native REST snapshot retries still lose updates to recreated fields**
   
   I reproduced this on `c36d679c` using a real REST catalog fixture and the 
Rust committer. I staged `age = 99` for an existing row, then dropped and 
re-added `age` and committed a competing append with `age = 12` as the first 
native publication request arrived.
   
   The append won the snapshot CAS. A single native commit call sent two 
publication requests, first with schema 0 and then with schema 1. The second 
request succeeded, but reading `age` returned `[NULL, 12]`.
   
   The new per-attempt validation covers the Python committer. Rust 
independently [loads the latest schema ID when building each 
snapshot](https://github.com/apache/paimon-rust/blob/e8939e5b419b380fd5da0d7be1092594907cee5e/crates/paimon/src/table/table_commit.rs#L1657-L1665),
 after the [Ray coordinator's field 
check](https://github.com/apache/paimon/blob/c36d679c33c08c75828b2b9928ec7d9eea856580/paimon-python/pypaimon/ray/row_id_conflict_rewriter.py#L111-L118)
 has already run. Please validate the staged field IDs against the schema used 
by each native publication attempt, including after a failed REST CAS.



##########
paimon-python/pypaimon/execution/python/write/file_store_commit.py:
##########
@@ -547,6 +548,12 @@ def _try_commit_once(self, retry_result: 
Optional[RetryResult], commit_kind: str
                 # Automatic rollback can delete snapshots and tags already 
used by readers.
                 raise conflict_exception
 
+        # Capture existing-row updates before assigning row IDs to new appends.
+        update_files = [
+            entry.file for entry in commit_entries
+            if entry.kind == 0 and entry.file.first_row_id is not None
+        ] if commit_kind == 'APPEND' and 
self.table.options.data_evolution_enabled() else []

Review Comment:
   **[P1] Validate staged updates when a mixed update/delete commits as 
OVERWRITE**
   
   A Ray self-merge can stage an UPDATE and a DELETE on different rows. With 
deletion vectors enabled, the DELETE makes the combined commit an `OVERWRITE`, 
so this condition leaves `update_files` empty.
   
   I reproduced this on `c36d679c` with real staged files and a real snapshot 
retry. I staged `age = 99` for row ID 0 and deleted row ID 1. During the first 
publication, I dropped and re-added `age` and committed a competing append. The 
first `OVERWRITE` attempt under schema 0 failed the CAS. The retry under schema 
1 succeeded. The delete took effect, but the remaining rows read back as IDs 
`[1, 3, 4]` with ages `[NULL, NULL, 10]`.
   
   Please select existing-row update files for validation even when deletion 
vectors make the commit an `OVERWRITE`, and add a mixed update/delete case to 
the inner CAS retry test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to