Omega359 opened a new pull request, #23772:
URL: https://github.com/apache/datafusion/pull/23772
## Which issue does this PR close?
- Part of #21706
## Rationale for this change
SQL benchmark suites expose configuration through environment variables.
Contributors must consult the source or documentation to discover suite-
specific settings, and Cargo does not forward custom arguments to
benchmark targets.
This change lets `benchmark_runner` expose those settings as validated
command-line arguments. It also adds suite-specific help and a dry-run mode for
inspecting resolved configuration without executing a benchmark.
## What changes are included in this PR?
This PR adds TOML `.suite` metadata files that define each SQL benchmark
suite’s:
- Description and help examples.
- Suite-specific options, defaults, environment variables, and accepted
values.
- Query filename pattern.
- Data-path replacements.
`benchmark_runner` uses this metadata to:
- Generate suite-specific command-line arguments and help output.
- Resolve suite options using command-line, environment-variable, and
default-value precedence.
- Support `--path` for suites that declare a `DATA_DIR` replacement.
- Support `--result-mode` for result persistence and validation.
- Validate suite names, metadata, option conflicts, query identifiers, and
incompatible arguments.
- Provide `--dry-run` JSON output containing the resolved options, paths,
value sources, filters, and execution mode without loading benchmark
definitions or executing SQL.
The existing basic runner, Criterion mode, and suite-listing functionality
remain in place.
## Are these changes tested?
Yes. New tests cover:
- Suite metadata parsing and validation.
- Dynamic suite options and help output.
- Option precedence between command-line arguments, environment variables,
and defaults.
- Path and query filename resolution.
- Dry-run output and validation.
- Result-mode resolution.
- Invalid option combinations and malformed metadata.
## Are there any user-facing changes?
Yes. Contributors can configure suite-specific settings through
command-line arguments and inspect them through suite help:
```shell
cargo run -p datafusion-benchmarks --release --bin benchmark_runner --
tpch --help
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- \
tpch --query 15 --format csv --path /path/to/tpch
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- \
clickbench --partitioning partitioned --dry-run
Environment variables remain supported for compatibility with direct cargo
bench invocations and existing benchmark definitions.
The SQL benchmark README documents the command-line options, resolution
precedence, dry-run behaviour, and .suite metadata format.
I have a script that cover a large number of combinations to exercise the
benchmark_runner as much as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
