Dandandan opened a new pull request, #23492: URL: https://github.com/apache/datafusion/pull/23492
## Which issue does this PR close? - None. This is an experimental performance enhancement with benchmark results included for review. ## Rationale for this change Parquet scans currently request the projected column chunks for each row group as the push decoder reaches it. For object stores with meaningful per-request latency, those sequential I/O requests can dominate query time even when the total amount of data is small. This change allows scans to batch I/O for complete subsequent row groups while keeping decoding lazy and enforcing a configurable byte budget. ## What changes are included in this PR? - Add `datafusion.execution.parquet.prefetch_size`, an optional byte budget that is disabled by default. - Use the push decoder's `buffered_bytes()` accounting when deciding how much speculative data can be staged. - Always fetch the ranges required by the current row group, even when they exceed the budget. - Append projected column ranges for complete subsequent row groups in scan order while `buffered + required + speculative` bytes fit the budget. - Keep prefetching I/O-only: prefetched row groups remain encoded in the push decoder until normal execution requests them. - Propagate the option through Parquet configuration and protobuf serialization. - Document the option and include the complete TPC-H and ClickBench per-query benchmark report in `benchmarks/parquet_prefetch_benchmark.md`. ### Benchmark results The benchmarks use three iterations, TPC-H SF1, a 167 MiB single-file ClickBench dataset, and the existing simulated S3-like latency wrapper (25-200 ms per GET). Speedup is prefetch-off time divided by 20 MiB prefetch time. | Configuration | TPC-H workload speedup | ClickBench workload speedup | |---|---:|---:| | Local I/O, filter pushdown off | 0.99x | 1.03x | | Simulated latency, filter pushdown off | 1.94x | 1.21x | | Local I/O, filter pushdown + reorder on | 1.02x | 1.01x | | Simulated latency, filter pushdown + reorder on | 1.73x | 1.18x | Increasing the latency-enabled budget from 20 MiB to 100 MiB did not improve the aggregate result: | Benchmark | Off | 20 MiB | 100 MiB | |---|---:|---:|---:| | TPC-H SF1 | 53082.86 ms | 30651.08 ms | 30688.45 ms | | ClickBench | 18852.91 ms | 15916.91 ms | 16100.72 ms | The full report contains setup details and every per-query timing. ## Are these changes tested? Yes. - `cargo fmt --all -- --check` - `cargo clippy --all-targets --all-features -- -D warnings` - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` - The prescribed extended workspace test command compiled and ran the Rust unit, integration, fuzz, and Parquet suites successfully. It identified the expected `df_settings` snapshot change introduced by the new option. - After updating that snapshot, the complete 493-file SQL logic suite passed. - Added focused push-decoder tests for budget packing, already-buffered byte accounting, disabled prefetch behavior, and staging prefetched row groups without another I/O request. - Benchmark query result row counts matched between prefetch-off and prefetch-on runs. ## Are there any user-facing changes? Yes. Users can opt in by setting `datafusion.execution.parquet.prefetch_size` to a byte budget such as `20971520` for 20 MiB. The default remains disabled, so existing behavior is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
