adriangb opened a new pull request, #25386: URL: https://github.com/apache/datafusion/pull/25386
## Which issue does this PR close? - N/A. This PR adds benchmarks only. It is split out of #25339 so that the suite is on `main` first, and that PR can then be measured against it. ## Rationale for this change A `NOT IN` subquery becomes a null-aware join. An outer row that finds no match is TRUE only when neither side has a NULL in scope. If a NULL is in scope, the result is UNKNOWN. This decision is cheap for an uncorrelated `NOT IN`. For a correlated `NOT IN`, the correlation predicate stays behind as a join filter. The join must then evaluate that filter for each candidate (build row x probe row) pair, to find which rows the NULLs reach. A non-equality correlation gives no equality key, so there is no scope key to reduce the number of pairs. The cost then grows with the NULL count multiplied by the size of the opposite table. No benchmark measured this shape, so there was no way to see the cost, or to tell a change from noise. Review on #25339 asked for this benchmark. These are the measured medians of 5 iterations, on a 4-core machine, in release mode, for `main` against #25339: | Query | Shape | main | #25339 | |---|---|---|---| | Q01 | uncorrelated, non-nullable keys | 59.9 ms | 68.3 ms | | Q02 | uncorrelated, 1% NULL subquery side | 50.7 ms | 53.1 ms | | Q03 | uncorrelated, 50% NULL outer side | 50.4 ms | 52.7 ms | | Q04 | correlated, nullable keys, no NULL present | 3.0 ms | 3.6 ms | | Q05 | correlated, 1% NULL outer side | 2.9 ms | 9.6 ms | | Q06 | correlated, 50% NULL outer side | 2.8 ms | 265.3 ms | | Q07 | correlated, 50% NULL subquery side | 2.7 ms | 265.8 ms | | Q08 | as Q06, with an equality correlation | 3.9 ms | 40.8 ms | `main` gives wrong results for Q05 to Q08, which is the bug that #25339 corrects. Thus those four rows show the cost of correct results, not a regression. Q01 to Q04 are the comparable rows. ## What changes are included in this PR? A `null_aware_join` SQL benchmark suite. There are no Rust changes. The runner finds suites in `benchmarks/sql_benchmarks/`, and the load SQL makes each table from `range()`, so there is no data generation step. - Q01 to Q03 are uncorrelated `NOT IN` at different NULL fractions. Their cost is linear with the table size. They are the regression guard for the plain null-aware path. - Q04 is the correlated shape with nullable keys that hold no NULL. It separates the baseline cost of the shape from the per-pair filter work. - Q05 to Q07 are the same correlation at 1% and 50% NULL on each side. This is where that work becomes visible. - Q08 has the same NULL fraction as Q06, but adds an equality correlation. The candidate pairs then come from a hash lookup. The difference between Q06 and Q08 shows the value of the scope key. Both table sizes are knobs. `NAJ_ROWS` (default 10000) sets the size for the correlated queries, whose cost grows with its square. `NAJ_LARGE_ROWS` (default 1000000) sets the size for the uncorrelated queries. ```bash ./bench.sh run null_aware_join # One query, with more rows for the correlated shape NAJ_ROWS=20000 ./bench.sh run null_aware_join 6 ``` This PR also adds the suite to `bench.sh` and documents it in `benchmarks/README.md` and `benchmarks/sql_benchmarks/README.md`. ## What is the testing strategy for this PR? This PR adds benchmarks, so it adds no new tests. The existing `checked_in_suites_cover_benchmark_directories` test in `benchmarks/src/sql_benchmark_suite.rs` covers suite discovery, and it passes with the new directory. All eight queries were run on this branch. Each one asserts `HashJoinExec` in its plan through the `expect_plan` directive. Each query also runs on `main` as written. Q08 uses the mark join form on purpose. The plain `WHERE ... NOT IN` form with an equality correlation does not plan on `main`, and a query that runs on only one branch cannot compare two branches. ## Are there any user-facing changes? No. This PR changes benchmarks and documentation only. It does not change library code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
