JingsongLi commented on code in PR #11269:
URL: https://github.com/apache/paimon/pull/11269#discussion_r4637836138
##########
paimon-python/pypaimon/read/read_dispatch.py:
##########
@@ -167,8 +167,9 @@ def to_arrow(
serially when a limit may discard rows, so discarded payloads
are not fetched by speculative split workers.
blob_parallelism: maximum concurrent blob range reads within each
- split reader. ``None`` or ``1`` (default) reads blobs serially;
- ``>= 2`` enables concurrent ranged reads. On the
+ split reader. ``None`` (default) allows up to 16 concurrent reads;
+ ``1`` reads blobs serially;
+ ``>= 2`` enables concurrent ranged reads. On the Python
Review Comment:
**[P2] The Python HTTP reader buffers a full object for each concurrent
range.** I tested one split containing 16 HTTP BLOB descriptors, each selecting
four bytes at a different offset in the same 2 MiB object. Both settings
returned the expected 64 bytes and issued 16 full GETs without a `Range`
header. With `blob_parallelism=1`, peak concurrent requests were 1 and
`tracemalloc` peaked at 5.8 MiB. With the new default, those figures rose to 16
requests and 30.5 MiB.
`HttpUriReader` loads `response.content` before `BlobRef` slices the
requested bytes. Could we send a byte-range request for descriptors with a
known offset and length? Until the HTTP path can bound the download, keeping
full-object HTTP reads serial by default would preserve the previous memory
behavior while allowing explicit overrides. [HTTP
reader](https://github.com/apache/paimon/blob/8d81a68a86d65441e9adb8522e1102d3aba31105/paimon-python/pypaimon/common/uri_reader.py#L72-L79)
##########
paimon-python/pypaimon/read/read_support.py:
##########
@@ -241,7 +241,7 @@ def _effective_parallelism(
def _resolve_blob_parallelism(runtime: Optional[int]) -> int:
if runtime is None:
- return 1
+ return 16
Review Comment:
**[P1] The new default can silently truncate BLOBs on the Python backend.**
I reproduced this with a single-split Data Evolution table containing two
100-byte BLOBs and a seekable input stream that returns at most 32 bytes per
read. Both `to_arrow(splits)` and `to_arrow_batch_reader(splits).read_all()`
returned payload lengths `[32, 0]` without raising. Passing
`blob_parallelism=1` to either API returned both original 100-byte payloads.
The serial reader loops until it has the indexed payload length. The
coalesced reader reads a span once and slices the returned buffer at the
original offsets, so a short read can also empty the next row. Please read each
fixed-length span to completion, advancing the offset after a short read and
raising on premature EOF. A regression test for both public APIs should compare
the default result with the original payloads and set `blob_parallelism=1`
explicitly for the serial control. [Coalesced range
reader](https://github.com/apache/paimon/blob/8d81a68a86d65441e9adb8522e1102d3aba31105/paimon-python/pypaimon/common/file_io.py#L379-L416)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]