XiaoHongbo-Hope commented on code in PR #9428:
URL: https://github.com/apache/paimon/pull/9428#discussion_r3878542331
##########
paimon-python/pypaimon/multimodal/blob_read.py:
##########
@@ -19,43 +19,82 @@
def fetch_blob_bodies(file_io, data, blob_cols, parallelism):
- """Fetch BLOB payload bytes for descriptor/inline/null cells.
+ """Fetch scalar and MAP BLOB payload bytes.
``data`` is a ``dict`` mapping each BLOB column name to row-aligned cells.
- Each cell may be serialized ``BlobDescriptor`` bytes, inline payload bytes,
- or ``None``. Returned values preserve row order and are grouped per column.
+ A cell may be serialized ``BlobDescriptor`` bytes, inline payload bytes,
+ ``None``, or a MAP represented by key-value pairs. Returned values preserve
+ row and MAP entry order and are grouped per column.
"""
from pypaimon.table.row.blob import BlobDescriptor, BlobViewStruct
ranges = []
inline = {}
- index = 0
+ targets = []
+ bodies = {col: [] for col in blob_cols}
+
+ def queue_blob_fetch(value, target):
+ index = len(ranges)
+ if value is None:
+ ranges.append(None)
+ else:
+ raw = bytes(value)
+ if BlobViewStruct.is_blob_view_struct(raw):
+ raise ValueError(
+ "read_blobs does not support unresolved blob-view columns;
"
+ "read such a column on its own, or enable blob-view
resolution.")
+ if BlobDescriptor.is_blob_descriptor(raw):
+ descriptor = BlobDescriptor.deserialize(raw)
+ ranges.append(
+ (descriptor.uri, descriptor.offset, descriptor.length)
+ )
+ else:
+ ranges.append(None)
+ inline[index] = raw
+ targets.append((target, index))
Review Comment:
Fixed in b81d34c91. Scalar BLOB columns now keep the contiguous offset/slice
path, while target tuples are created only for MAP entries. In the same 300k
inline-cell tracemalloc check, peak allocations dropped from 70.1 MiB to 25.1
MiB. The full multimodal table test suite passes (68 tests).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
