XiaoHongbo-Hope commented on code in PR #9239:
URL: https://github.com/apache/paimon/pull/9239#discussion_r3789656825
##########
paimon-python/pypaimon/write/table_update.py:
##########
@@ -228,17 +228,17 @@ def _update_by_predicate(
) -> List[CommitMessage]:
"""Shared implementation for SQL-like ``UPDATE ... WHERE ...``.
- ``predicate`` identifies the target rows. ``assignments`` maps target
- column names to literal values. The method reads matching ``_ROW_ID``
- values, builds an Arrow update table, then delegates to the existing
- row-id update path.
+ ``predicate`` identifies the target rows. Assignment values may be
+ literals or callables receiving the matched rows as an Arrow table.
"""
self._validate_predicate_update(assignments)
scan_table = self._matched_update_scan_table()
read_builder = scan_table.new_read_builder()
if predicate is not None:
read_builder.with_filter(predicate)
+ if predicate is not None or any(
+ callable(value) for value in assignments.values()):
Review Comment:
> **[P1] Avoid materializing every column for full-table callables**
>
> With `predicate=None`, any callable takes this branch, so even an
assignment that reads one scalar column projects every field and
`to_arrow(splits)` retains the whole table. Because `blob-as-descriptor`
defaults to false and BLOB deferral is disabled for predicate/limit-free reads,
this also resolves every BLOB payload before the callback runs. On
production-scale or multimodal tables this makes a single-column update require
unbounded driver memory and full BLOB I/O. Please require declared input
columns and execute bounded batches/ranges, or reject unbounded callable
updates.
Thanks. I also noticed this during development and opened #9240 to address
bounded processing separately. For this PR, I updated it as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
