andygrove opened a new pull request, #2048: URL: https://github.com/apache/datafusion-ballista/pull/2048
# Which issue does this PR close? Closes #1845. # Rationale for this change Ballista has a TPC-H correctness gate in CI (`--verify` cross-checks every query against single-process DataFusion over the same data), but no equivalent for TPC-DS. TPC-DS exercises a much wider surface — 24 tables, 99 queries with correlated subqueries, set operations, rollups, and window functions — so it reaches distributed-execution paths TPC-H does not. Adding a self-validating TPC-DS gate extends coverage with no golden-answer files to maintain. It already earned its keep: the initial SF1 sweep surfaced real distributed-execution bugs, now filed as #2046 (results diverge from single-process DataFusion on q4/q38/q78/q87 under the static planner) and #2047 (the adaptive planner panics with an `EmptyExec invalid partition` assertion on many queries). # What changes are included in this PR? - **Shared harness library** (`benchmarks/src/lib.rs`): the benchmark-agnostic result-comparison/oracle helpers are extracted from `benchmarks/src/bin/tpch.rs` and reused by both binaries. TPC-H behavior is unchanged. - **Data + queries**: `benchmarks/tpcds-gen.sh` generates SF1 TPC-DS Parquet via a git-pinned `tpcgen-cli`; `dev/vendor-tpcds-queries.sh` vendors the 99 DataFusion query files into `benchmarks/queries-tpcds/`. - **Runner** (`benchmarks/src/bin/tpcds.rs`): runs each non-skipped query on a Ballista cluster and, with `--verify`, diffs the result against a single-process DataFusion oracle over the same data. It records every per-query failure and exits non-zero if any query fails. - **CI** (`.github/workflows/tpcds.yml`): runs the gate at SF1 under the default (static) planner. Under the static planner the gate is green: 86/99 verified, 13 skipped. - **Skip list**: `SKIP` in `tpcds.rs` documents the excluded queries by cause — the #2046 divergences, genuinely non-deterministic queries (LIMIT/ORDER BY ties), and 5 queries whose `tpcgen-cli` schema column names differ from the DataFusion query text (not a Ballista issue). - **Docs**: a TPC-DS section in `benchmarks/README.md`. The adaptive planner (AQE on) is intentionally not run in CI yet because of #2047; re-adding it is a follow-up once that is fixed. Tested: `cargo fmt`/`clippy --all-targets --workspace -D warnings` clean; `ballista-benchmarks` tests pass (including the unchanged TPC-H `--verify` tests); the full SF1 gate runs green end-to-end on a native 1-scheduler/1-executor cluster (86 queries verified against DataFusion). # Are there any user-facing changes? A new `tpcds` benchmark binary, a new CI workflow, and benchmark documentation. No changes to public Ballista APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
