jayzhan211 commented on code in PR #27224: URL: https://github.com/apache/datafusion/pull/27224#discussion_r4601095994
########## datafusion/sqllogictest/test_files/rule_order_join_selection.slt: ########## @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements + +# This file runs `JoinSelection` after `EnsureRequirements` (see `moved_rule` in Review Comment: Yes. The PoC branch has a harness that runs the whole sqllogictest corpus under a rearranged rule list, including a seeded shuffle ([rule_order.rs](https://github.com/jayzhan211/datafusion/blob/poc/end-state-split/datafusion/sqllogictest/src/rule_order.rs)). It can't run in CI yet, because main still fails under many orders: 772 plan errors with `JoinSelection` after `EnsureRequirements` alone. So I replaced this file with unit tests, as you suggested, and each later rule fix will come with unit tests too. The shuffled corpus run goes into CI once the fixes have landed, together with #26152. Does that order work for you? ########## datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs: ########## @@ -806,6 +806,36 @@ fn new_join_conditions( .collect() } +/// Picks the mode of a hash join still in [`PartitionMode::Auto`]. +/// +/// An `Auto` join declares no distribution requirement, so its inputs would get +/// no repartitioning. `JoinSelection` normally picks the mode first. When it +/// hasn't (it runs later, or isn't in the rule list), use `Partitioned` if the +/// join may repartition its inputs and `CollectLeft` otherwise: enforcement can +/// satisfy either one. +pub(crate) fn resolve_auto_join_mode( Review Comment: Neither. The physical planner keeps emitting `Auto` as it does today, and there is no analyzer phase. `JoinSelection` picks the mode from statistics when it runs before `EnsureRequirements`, which is the default order, so default plans don't change. When it doesn't, `EnsureRequirements` resolves any join still in `Auto` (this PR), so no join is left in `Auto` in a rule list that contains `EnsureRequirements`. Running `JoinSelection` late then costs plan quality only: without statistics, a small build side is shuffled instead of collected. This replaces what I wrote on #26481 ("let the planner pick join modes"), which you asked about there: once enforcement resolves `Auto`, the physical planner doesn't need to. On the PoC branch, with the other rule fixes, the full corpus with `JoinSelection` after `EnsureRequirements` has no plan errors or wrong results (772 plan errors on main). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
