sunchao opened a new pull request, #27016: URL: https://github.com/apache/datafusion/pull/27016
## Which issue does this PR close? Part of #26682. ## Rationale for this change When each Parquet file has one row group, a reader can hide the next file's data request by starting it while decoding the current file. A benchmark that scans several partitions concurrently can hide the same delay even when each reader waits for its current file to finish. This gives #23492 and the cross-file experiment on #24086 a shared, single-partition workload. ## What changes are included in this PR? - Extend `sql_query_with_io` with a projected scan over 20 single-row-group files in one scan partition. The same SQL and bytes run against an in-memory store and a store with 100 ms of simulated read latency. Planning and data generation happen outside the timed loop. Each execution checks the sum of the selected column over 20 million rows. - Add a two-file object-store test. The full projection must read the selected column chunk once from each file and return all four values. With `LIMIT 1`, only one file's payload is needed. File statistics are disabled so the scan gets both files, and the assertions allow either file order. ## What is the testing strategy for this PR? `cargo test -p datafusion --test core_integration object_store_access` passed all 24 tests. The extended workspace suite passed 13,396 Rust tests and all 545 SQL logic files. Workspace formatting and Clippy with all targets and features passed. To reproduce the benchmark, run ```sh cargo bench -p datafusion --bench sql_query_with_io -- parquet_cross_file_projection ``` Local Criterion baseline on the PR branch, using a release build on a shared host, four pinned CPUs and ten samples per store. | Store | Estimated scan time | 95% confidence interval | | --- | ---: | ---: | | In memory | 24.6 ms | 24.3–24.9 ms | | 100 ms per read request | 2.059 s | 2.057–2.060 s | These two rows vary the store latency. They give future read-ahead changes a baseline under both conditions. ## Are there any user-facing changes? No. This adds benchmark and integration-test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
