gabotechs opened a new pull request, #23975:
URL: https://github.com/apache/datafusion/pull/23975
## Which issue does this PR close?
- None.
## Rationale for this change
Provide a repeatable way to measure how closely planning cardinality
statistics match runtime output for benchmark query suites.
## What changes are included in this PR?
`dfbench statistics` accepts any Parquet data directory and SQL file or
directory, so it can be used with TPC-DS, TPC-H, or another compatible query
suite. It:
- prints an indented, per-operator estimate-versus-runtime report as each
query completes;
- reports q-error per operator and finite q-error p50/p75/p95/p99 across the
run;
- distinguishes exact-zero estimates and infinite zero mismatches;
- continues after an individual query failure; and
- stores results by branch, comparing with the previous run by default or a
named branch using `--compare`.
### Example: TPC-DS query 3
```shell
cargo run --bin dfbench statistics \
--path benchmarks/data/tpcds_sf1 \
--query_path datafusion/core/tests/tpc-ds \
--query 3
```
```text
=== 3 ===
AggregateExec: rows=Inexact(1) vs 38, q-error: previous=?, current=38.00x,
change=?
HashJoinExec: rows=Inexact(1) vs 417, q-error: previous=?,
current=417.00x, change=?
DataSourceExec: rows=Inexact(18000) vs 18000, q-error: previous=?,
current=1.00x, change=?
=== q-error summary ===
current: branch 'my-branch'
comparison: previous run on branch 'my-branch'
queries: 1 succeeded, 0 failed
evaluated operators: 17/17
finite q-errors: 17, exact-zero estimates: 0, infinite q-errors: 0
p50: previous=?, current=19.00x, change=?
p75: previous=?, current=38.00x, change=?
p95: previous=?, current=417.00x, change=?
p99: previous=?, current=417.00x, change=?
```
### Compare against a baseline from `main`
```shell
# Run on main first; the result is stored under main
cargo run --bin dfbench statistics \
--path benchmarks/data/tpcds_sf1 \
--query_path datafusion/core/tests/tpc-ds \
--query 3
# Then, on a feature branch
cargo run --bin dfbench statistics \
--path benchmarks/data/tpcds_sf1 \
--query_path datafusion/core/tests/tpc-ds \
--query 3 \
--compare main
```
TPC-H works with its partitioned data layout as well:
```shell
cargo run --bin dfbench statistics \
--path benchmarks/data/tpch_sf1 \
--query_path benchmarks/queries \
--query q1
```
## Are these changes tested?
- `cargo check -p datafusion-benchmarks --bin dfbench`
- `cargo clippy -p datafusion-benchmarks --all-targets --no-deps -- -D
warnings`
- TPC-DS Q1 and Q3 smoke runs
- TPC-H Q1 smoke run
No unit tests were added: this is a reporting CLI over planner statistics
and runtime metrics, exercised by the benchmark-suite smoke runs above.
## Are there any user-facing changes?
Yes. This adds the `dfbench statistics` subcommand described above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
