JingsongLi commented on code in PR #9484:
URL: https://github.com/apache/paimon/pull/9484#discussion_r3891622866
##########
paimon-python/pypaimon/write/table_update.py:
##########
@@ -180,6 +181,15 @@ def _update_by_arrow_batches_with_row_id(
updater = TableUpdateByRowId(
self.table, self.commit_user, commit_identifier)
updater.update_columns(table, cols)
+ overlapping_first_row_ids = updated_first_row_ids.intersection(
Review Comment:
[P2] Detect overlap before staging the current batch
Could we perform this check before update_columns writes the current batch?
At this point, update_columns has already read and rewritten every targeted
file group. For the overlapping-batch case this change is intended to reject,
we therefore rewrite potentially large data or blob files only to abort them,
and cleanup is best-effort, so a failed deletion can leave orphan files. Please
split or extend update_columns so it calculates the first_row_id set, validates
it against the previously seen set, and only then writes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
