Akash3121 commented on code in PR #10113:
URL: https://github.com/apache/paimon/pull/10113#discussion_r4089745739
##########
paimon-python/pypaimon/read/reader/shard_batch_reader.py:
##########
@@ -38,8 +38,14 @@ def read_arrow_batch(self) -> Optional[RecordBatch]:
if isinstance(self.reader.format_reader, FormatBlobReader):
# For blob reader, pass begin_idx and end_idx parameters
return self.reader.read_arrow_batch(start_idx=self.start_pos,
end_idx=self.end_pos)
- else:
- # For non-blob reader (DataFileBatchReader), use standard
read_arrow_batch
+
+ # For non-blob reader (DataFileBatchReader), use standard
read_arrow_batch.
+ # Loop rather than recurse over skipped batches: a slice/shard whose
range
+ # sits deep in a file (default parquet batch_size is 1024 rows) skips
one
+ # batch per step, so recursing here overflows the stack
(RecursionError)
+ # once the skipped count exceeds the interpreter limit. Mirrors the
+ # while-loop skip pattern in ConcatBatchReader /
ApplyDeletionVectorReader.
+ while True:
Review Comment:
The loop removes the recursion failure, but it still drains every batch
after `end_pos` . Callers such as `TableRead.to_arrow()` read until this
method returns `None` , so for a `[0, 1)` slice of a 2,000-batch file, the
first call returns row 0 and the second call reads the remaining 1,999 batches
plus EOF. This means a head slice still scans the entire file, and an error in
data outside the selected range is surfaced even though that data should never
be read. Please return `None` before calling the underlying reader once
`current_pos >= end_pos` (or track an exhausted flag). Add a counting/raising
reader test for a large `[0, 1)` slice and assert that the second call
returns `None` without another underlying read.
Reproduction: a counting reader with 2,000 one-row batches required 1 call
to return `[0]` , then reached 2,001 total calls before the patched reader
returned `None` . A reader configured to throw on its second batch returned
the selected first row and then incorrectly propagated `OSError: tail should
not be read`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
