adriangb opened a new pull request, #23803:
URL: https://github.com/apache/datafusion/pull/23803
# Which issue does this PR close?
None — standalone benchmarking improvement, split out of the
bounded-streaming work (apache/arrow-rs#10410).
# Rationale for this change
The benchmark harness can simulate object-store latency
(`--simulate-latency`) but gives no visibility into how many requests a query
would issue against a real object store, which is the metric that determines
both cost and latency amplification of any change to the parquet fetch path.
# What changes are included in this PR?
- `CountingObjectStore`: registered under the (optional) latency wrapper for
every dfbench run; models S3 request semantics: one GET per `get_opts` call,
and one GET per *coalesced* range for `get_ranges` (1 MiB gap merge, following
`object_store::coalesce_ranges`), plus requested-byte accounting.
- Streamed local-file GET bodies are re-chunked at 2 MiB via
`object_store::chunked::ChunkedStore` (the raw local filesystem streams 8 KiB
chunks, which both misrepresents network granularity and is pathologically slow
for large streamed ranges).
- ClickBench / TPC-H / TPC-DS runners print per-iteration request and byte
counts:
`Query 20 iteration 0 took 587.4 ms and returned 110 rows (302 GET
requests, 913097581 bytes)`
Caveat: per-query request counts for TopK queries (e.g. ClickBench Q23) are
nondeterministic within a few requests because dynamic row-group pruning races
the scan.
# Are these changes tested?
Unit test for the coalescing model; exercised across ClickBench/TPC-H/TPC-DS
runs.
# Are there any user-facing changes?
Benchmark output lines gain request/byte counts. No engine changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01MGnT9oETcFMydc9cp95HLG
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
