andygrove opened a new pull request, #1966: URL: https://github.com/apache/datafusion-ballista/pull/1966
# Which issue does this PR close? <!-- No issue is filed for this; it adds test infrastructure. Happy to open one if preferred. --> N/A — adds plan-stability test coverage (analogous to Comet's TPC-DS plan-stability golden files). # Rationale for this change Ballista has no protection against accidental changes to the **distributed plan shape** of TPC-H queries. The existing per-query TPC-H tests (`benchmarks/src/bin/tpch.rs`) verify query *results*, not plan structure — a change that alters join strategy, shuffle/stage boundaries, stage count, or broadcast decisions but still returns correct rows passes silently. Comet (and Spark upstream) guard against exactly this with plan-stability golden files: each query's plan is frozen as a checked-in text file and CI fails when it drifts. This PR brings the same protection to Ballista for TPC-H. Concretely, it would have caught plan-shape changes such as a flip in the default join strategy or changes in exchange-reuse stage counts. # What changes are included in this PR? A new integration test in the `ballista-scheduler` crate under `ballista/scheduler/tests/tpch_plan_stability/`: - **Dataless stats-injecting table provider** (`stats_table.rs`): a `TableProvider` with a real TPC-H schema and injected SF100 row-count statistics (a `StatsExec` leaf modeled on DataFusion's `EmptyExec`), so the physical optimizer makes realistic join/broadcast choices without reading any data. - **Fixtures + helper** (`fixtures.rs`): the 8 TPC-H schemas, SF100 cardinalities, and `staged_plan_text()`, which plans a query with the static `DefaultDistributedPlanner` (`target_partitions=16`, Ballista default config), renders the resulting stages (`ShuffleWriterExec`/`UnresolvedShuffleExec` boundaries) to normalized text. - **Golden suite** (`main.rs`): `#[tokio::test]` cases `q1..q22` comparing the live staged plan against `approved/qN.txt`; regenerate with `BALLISTA_GENERATE_GOLDEN=1` (wrapped by `dev/update-tpch-plan-stability.sh`). - **22 approved plan files**, the query SQL copied under `queries/`, a `README.md`, and a RAT exclusion for the header-less golden text files. Scope (v1): TPC-H only, static planner, Ballista default config (SortMergeJoin). SHJ / adaptive variants can be added later as sibling golden directories. # Are there any user-facing changes? No. Test infrastructure only. Adds a developer helper script `dev/update-tpch-plan-stability.sh` for regenerating the approved plans after an intended change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
