mattsu2020 opened a new pull request, #26768: URL: https://github.com/apache/datafusion/pull/26768
## Which issue does this PR close? - Closes #21936. ## Rationale for this change Standard DataFusion environment settings can be silently overwritten by benchmark defaults. For example, `DATAFUSION_OPTIMIZER_PREFER_HASH_JOIN=false` is overridden by the legacy runners' default CLI value and by TPC-H's initialization SQL. ## What changes are included in this PR? - Share statistics and join configuration overrides across the TPC-H, TPC-DS, and IMDB runners through `QueryOpt`. Apply only explicitly supplied CLI values after `SessionConfig::from_env()`. - Remove duplicate environment aliases for session settings and remove TPC-H initialization SQL and `bench.sh` arguments that override those settings. - Document the standard environment variables and migration. Memory limits and pool types retain their separate `RuntimeEnv` configuration. ## What is the testing strategy for this PR? - Add subprocess coverage for standard environment settings, explicit CLI precedence, obsolete aliases, and consistency with the shared SQL runner. - Add `tpch_init_preserves_session_settings`. Restoring the previous TPC-H initialization SQL makes this test fail on the overwritten join setting; restoring the fix makes it pass. - `env -u DEBUG cargo test --profile ci -p datafusion-benchmarks --lib --bins --tests`: 235 tests passed. The inherited `DEBUG=release` is removed because it conflicts with the benchmark's boolean debug option. - `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`, `bash -n benchmarks/bench.sh`, and `git diff --check` passed. - Run the existing NLJ Q01 benchmark on upstream main (`350ca7650`) and this branch with two partitions and three iterations each. Both return 100,000 rows every time. These are unoptimized CI-profile smoke runs, not a performance claim. ## Are there any user-facing changes? Yes. Existing CLI flags remain available. Explicit CLI values take precedence over standard DataFusion environment settings, followed by DataFusion defaults. Scripts using these environment aliases must migrate: | Removed alias | Replacement | | --- | --- | | `PARTITIONS` | `DATAFUSION_EXECUTION_TARGET_PARTITIONS` | | `BATCH_SIZE` | `DATAFUSION_EXECUTION_BATCH_SIZE` | | `SORT_SPILL_RESERVATION_BYTES` | `DATAFUSION_EXECUTION_SORT_SPILL_RESERVATION_BYTES` | | `PREFER_HASH_JOIN` | `DATAFUSION_OPTIMIZER_PREFER_HASH_JOIN` | | `HASH_JOIN_BUFFERING_CAPACITY` | `DATAFUSION_EXECUTION_HASH_JOIN_BUFFERING_CAPACITY` | The benchmark READMEs include the updated configuration examples. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
