JingsongLi commented on code in PR #11048:
URL: https://github.com/apache/paimon/pull/11048#discussion_r4522713956


##########
paimon-python/pypaimon/ray/row_id_conflict_rewriter.py:
##########
@@ -413,7 +463,33 @@ def _partial_file_write_cols(table, file: DataFileMeta):
     )
     if schema is None:
         raise RuntimeError(f"Schema {file.schema_id} not found")
-    return schema.partial_file_write_cols(file.write_cols)
+    columns = schema.partial_file_write_cols(file.write_cols)
+    if columns is not None and any(SpecialFields.is_system_field(name) for 
name in columns):
+        return columns
+    current_fields = {field.id: field for field in table.table_schema.fields}
+    file_fields = {field.name: field for field in schema.fields}
+    fields = []
+    for name in columns if columns is not None else file_fields:
+        if name not in file_fields:
+            raise RuntimeError("Column '{}' not found in schema 
{}".format(name, file.schema_id))
+        fields.append(file_fields[name])
+    validate_row_id_update_fields(table.table_schema, table.schema_manager, 
[file])
+    # Resolve renamed fields by identity, including when an old name has been
+    # reused. TableRead applies the normal schema-evolution casts to the data.
+    return None if columns is None else [current_fields[field.id].name for 
field in fields]

Review Comment:
   **[P1] Preserve the historical ROW write scope when rebasing after schema 
evolution**
   
   I reproduced data loss on `8dfa8f74a` using real Java snapshots and Ray 
2.59.1. At snapshot 2 I staged `_ROW_ID=0` with `nest={a: 999}` under schema 0 
(`write_cols=['nest']`). Java then added `nest.b`, wrote `preserve-1` and 
`preserve-2` to `nest.b` in two rows, and compacted. The logical check 
considers the old and new leaf IDs disjoint. Recovery resolves the current 
top-level field here, reads the staged file using `nest(a,b)` at line 365, and 
submits a whole-ROW write at line 394. The retry committed snapshot 5 and both 
`nest.b` values became NULL, including the row outside the update input.
   
   Could the rewrite retain the original leaf-ID projection, or reject a ROW 
rewrite when the current type adds descendants that the historical file did not 
write? A Java partial-update fixture with a concurrent write to the added child 
would cover this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to