JingsongLi commented on code in PR #736:
URL: https://github.com/apache/paimon-rust/pull/736#discussion_r3844165759
##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -490,29 +490,36 @@ impl FormatFileReader for ParquetFormatReader {
// preserving positional `_ROW_ID`, sort order, and batch
backpressure. Reads
// with predicates or an explicit row selection retain the original
// single-stream path until their selections are split per row group.
- let row_group_parallelism = self
- .read_budget
- .as_ref()
- .filter(|_| preds.is_empty() && row_filter_factory.is_none() &&
row_selection.is_none())
+ let read_budget = self.read_budget.as_ref().filter(|_| {
+ preds.is_empty() && row_filter_factory.is_none() &&
row_selection.is_none()
+ });
+ let row_group_parallelism = read_budget
.map(|budget| {
budget
.parallelism()
.min(batch_stream_builder.metadata().num_row_groups())
})
.unwrap_or(1);
+ let projected_bytes = read_budget
Review Comment:
Because `read_budget` has already been filtered by
`row_selection.is_none()`, any partial-row-range read reaches this block with
`read_budget == None`. Vector-index shard boundaries can cut through a file, so
these reads still perform Parquet I/O while reporting `parquet_row_group_count`
and all projected-byte diagnostics as zero. Please compute and record the
diagnostic sizes from `self.read_budget` independently of parallel-path
eligibility, or rename these fields to make their parallel-path-only scope
explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
