YannByron commented on code in PR #9445:
URL: https://github.com/apache/paimon/pull/9445#discussion_r3886832613
##########
paimon-python/pypaimon/write/table_update.py:
##########
@@ -166,6 +166,26 @@ def _update_by_arrow_with_row_id(
self.table, self.commit_user, commit_identifier,
).update_columns(table, cols)
+ def _update_by_arrow_batches_with_row_id(
+ self, tables: Iterable[pa.Table], commit_identifier: int
+ ) -> List[CommitMessage]:
+ updater = None
+ try:
+ for table in tables:
+ cols = self.update_cols if self.update_cols is not None else [
+ c for c in table.column_names
+ if c != SpecialFields.ROW_ID.name
+ ]
+ if updater is None:
+ updater = TableUpdateByRowId(
+ self.table, self.commit_user, commit_identifier)
+ updater.update_columns(table, cols)
Review Comment:
Thanks for reproducing this. In the current RoboMIND backfill path, updates
come directly from Paimon planned splits. DataEvolutionSplitGenerator groups
files by first_row_id before packing splits, and _iter_raw materializes each
complete split as one batch, so the same first_row_id group is not expected to
cross batches in this specific flow. However, you are right that the general
batch API neither expresses nor enforces this precondition, so arbitrary or
concurrently produced batches can hit this correctness issue. The ideal general
solution is to determine first_row_id before writing, shuffle or partition
updates by it, coalesce all updates for the same target file, write one delta
per group, and then perform one coordinated commit. That is broader
distributed-update work. For this PR, I will add exact overlap detection for
the affected first_row_ids and fail before commit, aborting staged files
instead of silently losing updates. I will also add the same-file end-to-end
test.
I plan to address true concurrent cross-batch updates with shuffle and
coalescing in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
