andygrove opened a new pull request, #2799: URL: https://github.com/apache/datafusion-ballista/pull/2799
# Which issue does this PR close? No single issue. Split out of #2792. Related: #2520 (performance review tracking), #1375 (dynamic filters). **Depends on the semi-join pushdown PR (#2793).** It is built on that PR's commit, so review only the last commit; this PR will be rebased once the first one merges. # Rationale for this change DataFusion applies a selective dimension filter (such as `nation`) only at the join that names it. TPC-H q7 joins and shuffles 182M `lineitem` rows before the nation filter drops 92% of them. Spark's runtime filters avoid this. # What changes are included in this PR? A rule, `SemiJoinReduction`, controlled by `ballista.optimizer.semi_join_reduction` (default `true`). For an inner join against a dimension filtered by `=`, `IN` or `LIKE` on literals, it adds a semi join against a copy of the dimension on the table the key comes from, below the joins in between. In q7, `supplier` and `customer` are reduced to the two filtered nations before they join `lineitem`. Logical plans have no statistics, so the scheduler runs the rule through `apply_semi_join_reduction` in `TaskManager::submit_job` before physical planning: - Candidates are found by shape, then planned on their own to read their row estimate. Only dimensions under `ballista.optimizer.broadcast_join_threshold_rows` are kept. - A semi join that lands on a table under 100K estimated rows (`nation` reduced by `region` in q2/q5/q8) only adds a join, so it is removed again. - The semi joins sink through the shared `step` helper in `SemiJoinPushdown`, but, unlike that rule, also through joins with a filtered input, since the semi join is the selective side there. With every change in #2792 together, on one machine at TPC-H SF100 (2 executors x 8 cores, 16 partitions), q7 goes from 20.1 s to 5.6 s. All 22 queries pass `--verify` against DataFusion at SF10 on that branch. # Are there any user-facing changes? One new config key, `ballista.optimizer.semi_join_reduction`, enabled by default. Plans for queries with filtered dimension joins change. `docs/source/user-guide/configs.md` is regenerated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
