JingsongLi commented on code in PR #11044:
URL: https://github.com/apache/paimon/pull/11044#discussion_r4523239732
##########
paimon-python/pypaimon/multimodal/columns.py:
##########
@@ -240,6 +244,55 @@ def merge_columns(owner, data, *, on):
return operation.finish(None, commit=commit, messages=messages)
+def update_columns_by_row_id(owner, data, *, columns):
+ if not isinstance(data, pa.Table):
+ raise ValueError("data must be a pyarrow.Table.")
+ raw = owner.raw_table
+ _validate_update(raw)
+ if not raw.options.data_evolution_enabled() or not
raw.options.row_tracking_enabled():
+ raise ValueError("Row ID column updates require a Data Evolution table
with row tracking.")
+ row_id = SpecialFields.ROW_ID.name
+ if data.column_names.count(row_id) != 1:
+ raise ValueError("Input must contain exactly one _ROW_ID column.")
+ columns = [columns] if isinstance(columns, str) else list(columns)
+ _validate_names(columns, "Output")
+ missing = set(columns).difference(raw.field_names)
+ if missing:
+ raise ValueError("Output columns do not exist: %s" % sorted(missing))
+ if set(columns).intersection(raw.partition_keys):
+ raise ValueError("update_columns_by_row_id cannot update partition
columns.")
+ if set(columns).intersection(raw.options.video_frame_fields()):
+ raise ValueError("Use replace_video() to update video-frame columns.")
+ for name in columns:
+ if data.column_names.count(name) != 1:
+ raise ValueError("Input must contain exactly one %s column." %
name)
+ row_ids = normalize_row_ids(data[row_id])
+ if len(pc.unique(row_ids)) != data.num_rows:
+ raise ValueError("Input data contains duplicate _ROW_ID values.")
+ if data.num_rows and pc.min(row_ids).as_py() < 0:
+ raise ValueError("_ROW_ID values must be non-negative.")
+ target_schema = _target_schema(raw)
Review Comment:
**[P2] Reject a stale schema before reporting successful row-ID updates**
I kept a normal `conn.get_table()` handle, then dropped and re-added its
`label` column with the same type. That DDL completed before the call and did
not advance the data snapshot. Calling `update_columns_by_row_id(...,
columns=["label"])` with the old handle returned `updated_rows=1`,
`schema_id=0`, and a new snapshot. A fresh schema-1 reader still returned NULL
for `label`. The row-ID check uses the latest data snapshot, but the write
resolves the old field ID from the cached schema. Could we check the current
schema before staging this operation and reject or refresh a stale handle so
the result cannot report a write that current readers never see?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]