viirya commented on code in PR #11165:
URL: https://github.com/apache/arrow-rs/pull/11165#discussion_r4074088753
##########
parquet/src/arrow/push_decoder/mod.rs:
##########
@@ -605,6 +619,36 @@ impl ParquetPushDecoder {
self.state.peek_next_row_group()
}
+ /// Preview at most two filter-free row groups using the demand range
planner.
+ ///
+ /// Returns `None` outside a row-group boundary or when row predicates are
+ /// present; returns an empty vector when there is no selected work. This
+ /// never advances the decoder, evaluates predicates, reads data, or
creates
+ /// batch readers. Ranges fully covered by an input buffer are excluded;
Review Comment:
My wording was unclear. The preview does not simulate buffer consumption
between entries; each entry checks the same current buffers.
For example, suppose the plan visits row group B twice and its requested
ranges are already buffered. A two-entry preview reports empty ranges for both
visits. During actual execution, the first visit can remove those buffers, so
the second visit subsequently returns `NeedsData` for B.
The required file ranges have not changed—only whether they are still
buffered. This is intentional and covered by
`read_ahead_preview_preserves_duplicate_row_groups`; my suggestion was to make
that distinction explicit in the public docs.
Your example of an existing object-store cache is also why I asked about
exposing required ranges independently of decoder buffer coverage: the caller
could then use its own cache to decide which reads to issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
