alamb commented on code in PR #23772:
URL: https://github.com/apache/datafusion/pull/23772#discussion_r3723117526
##########
benchmarks/sql_benchmarks/README.md:
##########
@@ -43,24 +43,96 @@ in the community:
| `tpcds` | TPC‑DS queries
|
| `tpch` | TPC‑H queries
|
| `wide_schema` | Small-projection queries on a wide (1024-col,
256-file) synthetic dataset; runs `wide` + `narrow` subgroups for comparison |
-| `predicate_eval` | Conjunctive (AND) filter-evaluation
micro-benchmarks; each subgroup is a different predicate pattern, to test how
an adaptive predicate-ordering system behaves across them
([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups
(`BENCH_SUBGROUP`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`,
`scale`, `neutral`, `correlation`, `drift`. Toggle a system under test with its
native `DATAFUSION_*` env var |
+| `predicate_eval` | Conjunctive (AND) filter-evaluation
micro-benchmarks; each subgroup is a different predicate pattern, to test how
an adaptive predicate-ordering system behaves across them
([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups
(`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`,
`scale`, `neutral`, `correlation`, `drift`. Configure the system under test
through its DataFusion settings. |
# Running Benchmarks
-The easiest way to run a benchmark is to use the `bench.sh` shell script (up
one level from this document)
-as it takes care of configuring any required environment variables and can
populate any required data files.
-However, it is possible to directly run a sql benchmark using the `cargo
bench` command. For example:
+Use `benchmark_runner` to run SQL benchmarks. It reads each suite's `.suite`
+file and exposes the suite's configuration as command-line options. Use the
+`bench.sh` shell script one level above this directory to download or generate
+required data files.
```shell
-BENCH_NAME=tpch cargo bench --bench sql
+cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch
```
+## SQL benchmark runner
+
+The `benchmark_runner` binary discovers suites from this directory and exposes
+suite-specific options alongside the common benchmark options. The suite name
+must come before all options.
+
+```bash
+cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- --list
+cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch
--help
+cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch
--query 15 --format csv
+cargo run -p datafusion-benchmarks --release --bin benchmark_runner --
clickbench --partitioning partitioned --dry-run
+cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch
--query 1 --result-mode persist
+cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch
--query 1 --result-mode validate
Review Comment:
it is really cool
(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/runner$ cargo run -p
datafusion-benchmarks --profile=profiling --bin benchmark_runner -- h2o
--size=medium --subgroup=groupby
Finished `profiling` profile [optimized + debuginfo] target(s) in 0.16s
Running `target/profiling/benchmark_runner h2o --size=medium
--subgroup=groupby`
Loading medium groupby csv h2o data
h2o/Q01/groupby iteration 0: 895.4 ms, 100 rows
h2o/Q01/groupby iteration 1: 886.0 ms, 100 rows
h2o/Q01/groupby iteration 2: 884.8 ms, 100 rows
Loading medium groupby csv h2o data
##########
benchmarks/sql_benchmarks/clickbench/clickbench.suite:
##########
@@ -0,0 +1,25 @@
+description = "ClickBench analytics queries over the hits dataset"
+
+query_pattern = "q{QUERY_ID_PADDED}.benchmark"
+
+[path_replacements]
+DATA_DIR = "../../data"
+
+[[options]]
+name = "partitioning"
+env = "CLICKBENCH_TYPE"
+default = "single"
+values = ["single", "partitioned"]
+help = "Selects the single-file or partitioned ClickBench dataset."
+
+[[examples]]
+command = "cargo run --release --bin benchmark_runner -- clickbench"
+description = "Run all ClickBench queries against the single-file dataset."
+
+[[examples]]
+command = "cargo run --release --bin benchmark_runner -- clickbench --query 7"
Review Comment:
I tried it like
```shell
cargo run -p datafusion-benchmarks --release --bin benchmark_runner --
clickbench --query 7
```
```shell
(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/runner$ cargo run -p
datafusion-benchmarks --release --bin benchmark_runner -- clickbench --query 7
Finished `release` profile [optimized] target(s) in 0.24s
Running `target/release/benchmark_runner clickbench --query 7`
clickbench/Q07 iteration 0: 146.3 ms, 18 rows
clickbench/Q07 iteration 1: 28.0 ms, 18 rows
clickbench/Q07 iteration 2: 22.9 ms, 18 rows
```
It was also sweet to be able to do
```shell
cargo run -p datafusion-benchmarks --release --bin benchmark_runner --
clickbench --query 7 --iterations=100
```
And have it do
```shell
(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/runner$ cargo run -p
datafusion-benchmarks --release --bin benchmark_runner -- clickbench --query 7
--iterations=100
Finished `release` profile [optimized] target(s) in 0.21s
Running `target/release/benchmark_runner clickbench --query 7
--iterations=100`
clickbench/Q07 iteration 0: 37.4 ms, 18 rows
clickbench/Q07 iteration 1: 28.1 ms, 18 rows
...
clickbench/Q07 iteration 96: 29.0 ms, 18 rows
clickbench/Q07 iteration 97: 26.2 ms, 18 rows
clickbench/Q07 iteration 98: 29.2 ms, 18 rows
clickbench/Q07 iteration 99: 26.4 ms, 18 rows
```
##########
benchmarks/sql_benchmarks/clickbench_extended/clickbench_extended.suite:
##########
@@ -0,0 +1,21 @@
+description = "Extended ClickBench queries over the hits dataset"
+
+query_pattern = "q{QUERY_ID_PADDED}.benchmark"
+
+[path_replacements]
+DATA_DIR = "../../data"
+
+[[options]]
+name = "partitioning"
Review Comment:
this is pretty neat -- so I can do
You can easily see the diiffernc
```shell
(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/runner$ cargo run -p
datafusion-benchmarks --release --bin benchmark_runner -- clickbench_extended
--iterations=1
Finished `release` profile [optimized] target(s) in 0.40s
Running `target/release/benchmark_runner clickbench_extended
--iterations=1`
clickbench_extended/Q00/single iteration 0: 616.4 ms, 1 rows
clickbench_extended/Q01/single iteration 0: 120.4 ms, 1 rows
clickbench_extended/Q02/single iteration 0: 263.7 ms, 10 rows
clickbench_extended/Q03/single iteration 0: 243.9 ms, 10 rows
clickbench_extended/Q04/single iteration 0: 1247.9 ms, 2 rows
clickbench_extended/Q05/single iteration 0: 9762.0 ms, 2 rows
clickbench_extended/Q06/single iteration 0: 11.7 ms, 1 rows
clickbench_extended/Q07/single iteration 0: 520.7 ms, 10 rows
clickbench_extended/Q08/single iteration 0: 294.6 ms, 10 rows
clickbench_extended/Q09/single iteration 0: 1907.5 ms, 1 rows
clickbench_extended/Q10/single iteration 0: 576.0 ms, 1 rows
clickbench_extended/Q11/single iteration 0: 1140.2 ms, 1 rows
clickbench_extended/Q12/single iteration 0: 129.8 ms, 1 rows
```
And
```
(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/runner$ cargo run -p
datafusion-benchmarks --release --bin benchmark_runner -- clickbench_extended
--iterations=1 --partitioning partitioned
Finished `release` profile [optimized] target(s) in 0.40s
Running `target/release/benchmark_runner clickbench_extended
--iterations=1 --partitioning partitioned`
clickbench_extended/Q00/partitioned iteration 0: 651.8 ms, 1 rows
clickbench_extended/Q01/partitioned iteration 0: 102.8 ms, 1 rows
clickbench_extended/Q02/partitioned iteration 0: 237.8 ms, 10 rows
clickbench_extended/Q03/partitioned iteration 0: 226.6 ms, 10 rows
clickbench_extended/Q04/partitioned iteration 0: 1289.1 ms, 2 rows
clickbench_extended/Q05/partitioned iteration 0: 10181.2 ms, 2 rows
clickbench_extended/Q06/partitioned iteration 0: 4.7 ms, 1 rows
clickbench_extended/Q07/partitioned iteration 0: 716.6 ms, 10 rows
clickbench_extended/Q08/partitioned iteration 0: 281.5 ms, 10 rows
clickbench_extended/Q09/partitioned iteration 0: 1800.9 ms, 1 rows
clickbench_extended/Q10/partitioned iteration 0: 457.3 ms, 1 rows
clickbench_extended/Q11/partitioned iteration 0: 892.2 ms, 1 rows
clickbench_extended/Q12/partitioned iteration 0: 117.3 ms, 1 rows
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
