XiaoHongbo-Hope commented on code in PR #9259:
URL: https://github.com/apache/paimon/pull/9259#discussion_r3812683891
##########
paimon-python/pypaimon/data/variant_path.py:
##########
@@ -1976,8 +2228,75 @@ def rebuild_row(row, view, insert_set, key_ids,
names_by_id,
new_metadata, key_ids, names_by_id = _metadata_with_keys(
first_metadata, insert_keys)
insert_set = set(insert_indices)
+ splice_eligible = (
+ len(insert_indices) == 1
+ and len(parsed[insert_indices[0]][1]) == 1
+ and all(
+ index in insert_set
+ or parsed[index][2]._fixed_size is not None
+ for index in range(count))
+ )
+ if splice_eligible:
+ insert_index = insert_indices[0]
+ replace_indices = [
+ index for index in range(count) if index != insert_index
+ ]
+ if replace_indices:
+ _patch_planned_group(
+ (rows, row_starts, source_data,
+ [target_positions[index] for index in replace_indices],
+ [target_limits[index] for index in replace_indices]),
+ [parsed[index] for index in replace_indices],
+ len(chunk), global_row, state, group_slow, False)
+ slow_rows |= group_slow
+ if group_slow:
+ keep = np.fromiter(
+ (int(row) not in group_slow for row in rows),
+ bool, len(rows))
+ live_rows = rows[keep]
+ live_starts = row_starts[keep]
+ live_lengths = parent_limits[insert_index][keep]
+ else:
+ live_rows = rows
+ live_starts = row_starts
+ live_lengths = parent_limits[insert_index]
+ provider = parsed[insert_index][2]
+ if provider._array is None:
+ payloads = [
+ payload_for(insert_index, provider, 0)
+ ] * len(live_rows)
+ else:
Review Comment:
> [P1] Avoid retaining the entire variable-width payload column here
>
> For an array-backed string or binary insert, this list eagerly encodes
every live row. `_root_insert_splice` then keeps the list alive while
accumulating every rebuilt row, so peak memory grows by roughly one extra copy
of the whole inserted payload. I reproduced this with 50,000 rows x 1 KiB: max
RSS increased from 370.8 MB on the base to 440.9 MB on this head (+70.1 MB); an
independent 100,000-row run showed +134.9 MB. Large production chunks can
therefore OOM even though the NumPy index matrix is bounded. Please stream or
byte-budget the encoding/splice work, and add a peak-memory regression test,
instead of materializing the entire payload list.
Thanks, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
