Omega359 opened a new pull request, #24803:
URL: https://github.com/apache/datafusion/pull/24803
## Which issue does this PR close?
Part of #21706. This PR does not close the issue.
## Rationale for this change
The SQL-based TPC-H, TPC-DS, and sort TPC-H benchmarks did not
consistently expose the same schemas, constraints, and loading behavior as
their native Rust equivalents. These differences could affect query planning
and make benchmark results less directly comparable.
## What changes are included in this PR?
- Align the TPC-H CSV, Parquet, and in-memory table definitions with the
native benchmark's data types, nullability, and primary-key constraints.
- Add explicit TPC-DS schemas and primary-key constraints while retaining
the physical types used by the DataFusion benchmark Parquet dataset.
- Update the sort TPC-H loader to:
- Read the complete `lineitem` Parquet directory rather than only
`lineitem.1.parquet`.
- Keep the table Parquet-backed instead of materializing it.
- Declare existing `l_orderkey ASC NULLS LAST` ordering as metadata
instead of physically sorting the data.
The sort TPC-H changes match the default behavior of the native Rust
benchmark when it is run without `--mem-table`.
## Are these changes tested?
Yes.
The following checks were run:
```shell
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
Sort TPC-H query 1 was also executed against the SF1 dataset in both modes:
cargo run --profile release -p datafusion-benchmarks \
--bin benchmark_runner -- sort_tpch \
--scale-factor 1 --sorted false --query 1 --iterations 1
cargo run --profile release -p datafusion-benchmarks \
--bin benchmark_runner -- sort_tpch \
--scale-factor 1 --sorted true --query 1 --iterations 1
Both modes completed successfully and returned all 6,001,215 lineitem rows.
## Are there any user-facing changes?
No public API behavior changes. The SQL benchmark definitions now more
closely match the corresponding native benchmark configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
