JingsongLi commented on code in PR #8488:
URL: https://github.com/apache/paimon/pull/8488#discussion_r3534151575
##########
paimon-python/pypaimon/read/reader/blob_descriptor_convert_reader.py:
##########
@@ -187,6 +189,39 @@ def _resolve_descriptor_fields(self, batch):
pyarrow.field(field_name, pyarrow.large_binary(),
nullable=True),
pyarrow.array(converted_values, type=pyarrow.large_binary()),
)
+
+ view_file_ios = view_file_ios or {}
+ for field_name in self._view_fields:
+ field_file_ios = view_file_ios.get(field_name)
+ if field_name not in batch.schema.names or field_file_ios is None:
+ continue
+ values = [self._normalize_blob_to_bytes(v) for v in
batch.column(field_name).to_pylist()]
+ blobs_by_file_io = {}
+ converted_values = []
+
+ for idx, value in enumerate(values):
+ file_io = field_file_ios[idx] or self._table.file_io
+ blob = Blob.from_bytes(value, file_io)
Review Comment:
This still goes through `Blob.from_bytes`, whose descriptor branch calls
`file_io.uri_reader_factory.create(descriptor.uri)` instead of reading through
the `file_io` object passed here. For BlobView fields that means the default
non-parallel path can still rebuild a reader and lose the upstream FileIO
context/credentials (for example `RESTTokenFileIO`), while the parallel path
below uses `file_io.read_blobs_concurrent(...)` and does use the cached
upstream FileIO. Could we construct the view-field `BlobRef` from this
`file_io` directly here (or merge the `Blob.from_bytes` fix first) and add a
test with an upstream FileIO whose factory must not be used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
