JingsongLi commented on code in PR #11269:
URL: https://github.com/apache/paimon/pull/11269#discussion_r4637836138


##########
paimon-python/pypaimon/read/read_dispatch.py:
##########
@@ -167,8 +167,9 @@ def to_arrow(
             serially when a limit may discard rows, so discarded payloads
             are not fetched by speculative split workers.
         blob_parallelism: maximum concurrent blob range reads within each
-            split reader. ``None`` or ``1`` (default) reads blobs serially;
-            ``>= 2`` enables concurrent ranged reads. On the
+            split reader. ``None`` (default) allows up to 16 concurrent reads;
+            ``1`` reads blobs serially;
+            ``>= 2`` enables concurrent ranged reads. On the Python

Review Comment:
   **[P2] The Python HTTP reader buffers a full object for each concurrent 
range.** I tested one split containing 16 HTTP BLOB descriptors, each selecting 
four bytes at a different offset in the same 2 MiB object. Both settings 
returned the expected 64 bytes and issued 16 full GETs without a `Range` 
header. With `blob_parallelism=1`, peak concurrent requests were 1 and 
`tracemalloc` peaked at 5.8 MiB. With the new default, those figures rose to 16 
requests and 30.5 MiB.
   
   `HttpUriReader` loads `response.content` before `BlobRef` slices the 
requested bytes. Could we send a byte-range request for descriptors with a 
known offset and length? Until the HTTP path can bound the download, keeping 
full-object HTTP reads serial by default would preserve the previous memory 
behavior while allowing explicit overrides. [HTTP 
reader](https://github.com/apache/paimon/blob/8d81a68a86d65441e9adb8522e1102d3aba31105/paimon-python/pypaimon/common/uri_reader.py#L72-L79)



##########
paimon-python/pypaimon/read/read_support.py:
##########
@@ -241,7 +241,7 @@ def _effective_parallelism(
 
 def _resolve_blob_parallelism(runtime: Optional[int]) -> int:
     if runtime is None:
-        return 1
+        return 16

Review Comment:
   **[P1] The new default can silently truncate BLOBs on the Python backend.** 
I reproduced this with a single-split Data Evolution table containing two 
100-byte BLOBs and a seekable input stream that returns at most 32 bytes per 
read. Both `to_arrow(splits)` and `to_arrow_batch_reader(splits).read_all()` 
returned payload lengths `[32, 0]` without raising. Passing 
`blob_parallelism=1` to either API returned both original 100-byte payloads.
   
   The serial reader loops until it has the indexed payload length. The 
coalesced reader reads a span once and slices the returned buffer at the 
original offsets, so a short read can also empty the next row. Please read each 
fixed-length span to completion, advancing the offset after a short read and 
raising on premature EOF. A regression test for both public APIs should compare 
the default result with the original payloads and set `blob_parallelism=1` 
explicitly for the serial control. [Coalesced range 
reader](https://github.com/apache/paimon/blob/8d81a68a86d65441e9adb8522e1102d3aba31105/paimon-python/pypaimon/common/file_io.py#L379-L416)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to