sesteves opened a new issue, #24527: URL: https://github.com/apache/datafusion/issues/24527
## Describe the bug After [PR #20839](https://github.com/apache/datafusion/pull/20839) replaced the pull-based Parquet reader with `ParquetPushDecoder`, a controlled comparison found substantially more object-store requests for a large filtered Parquet scan. The leading explanation is that predicate and projected-column ranges are discovered in separate `NeedsData` rounds. DataFusion can coalesce ranges within one `get_byte_ranges` call, but not with ranges discovered in a later decoder round. Current `main` still follows this sequence: ```text try_next_reader() -> NeedsData(ranges) -> get_byte_ranges(ranges).await -> push_ranges(...) -> repeat ``` In architectures where compute and storage are decoupled, each additional object-store round trip adds directly to scan latency. This is especially significant when scans touch many small Parquet files. The request-level attribution to predicate versus projection rounds is inferred from the source and the A/B results below. The additional requests themselves are directly measured. ## To reproduce Compare a selective filter with a narrow projection over many remote Parquet files while instrumenting `AsyncFileReader` request counts. Our controlled comparison upgraded only DataFusion from 53.1.0 to 54.1.0 while retaining `parquet` 58.3.0 and `object_store` 0.13.2: | Configuration | Object-store requests | Bytes read | |---|---:|---:| | DataFusion 53.1.0 | 708K-709K | ~139 GB | | DataFusion 54.1.0 with filter pushdown | 894K-898K | ~86.6 GB | | DataFusion 54.1.0 without filter pushdown | 583K | ~186.3 GB | The DF54 request increase closely matched one additional request per filtered file. Disabling filter pushdown is not a viable workaround. It reduces request count but more than doubles bytes read and produced a 164-second runtime in this workload. ## Expected behavior Retain the byte savings from predicate pushdown without requiring a separate network round trip for predicate and projected-column ranges. Possible directions include: - Coalesce ranges across consecutive decoder rounds. - Speculatively fetch projected-column ranges where appropriate. - Extend the arrow-rs decoder API to expose upcoming optional ranges. - Add request-count coverage using a counting, delayed object store. ## Additional context - PR #20839 explicitly identified coalescing and prefetching as goals of the push-decoder migration. - Closed [PR #21370](https://github.com/apache/datafusion/pull/21370) coalesced adjacent ranges within one `NeedsData` result, but did not address ranges discovered across separate rounds. - A search did not find an existing DataFusion issue covering this cross-round request amplification. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
