jayzhan211 commented on code in PR #27224:
URL: https://github.com/apache/datafusion/pull/27224#discussion_r4601095994


##########
datafusion/sqllogictest/test_files/rule_order_join_selection.slt:
##########
@@ -0,0 +1,51 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under 
one or more contributor license agreements
+
+# This file runs `JoinSelection` after `EnsureRequirements` (see `moved_rule` 
in

Review Comment:
   Yes. The PoC branch has a harness that runs the whole sqllogictest corpus 
under a rearranged rule list, including a seeded shuffle 
([rule_order.rs](https://github.com/jayzhan211/datafusion/blob/poc/end-state-split/datafusion/sqllogictest/src/rule_order.rs)).
 It can't run in CI yet, because main still fails under many orders: 772 plan 
errors with `JoinSelection` after `EnsureRequirements` alone. So I replaced 
this file with unit tests, as you suggested, and each later rule fix will come 
with unit tests too. The shuffled corpus run goes into CI once the fixes have 
landed, together with #26152. Does that order work for you?
   



##########
datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:
##########
@@ -806,6 +806,36 @@ fn new_join_conditions(
         .collect()
 }
 
+/// Picks the mode of a hash join still in [`PartitionMode::Auto`].
+///
+/// An `Auto` join declares no distribution requirement, so its inputs would 
get
+/// no repartitioning. `JoinSelection` normally picks the mode first. When it
+/// hasn't (it runs later, or isn't in the rule list), use `Partitioned` if the
+/// join may repartition its inputs and `CollectLeft` otherwise: enforcement 
can
+/// satisfy either one.
+pub(crate) fn resolve_auto_join_mode(

Review Comment:
   Neither. The physical planner keeps emitting `Auto` as it does today, and 
there is no analyzer phase. `JoinSelection` picks the mode from statistics when 
it runs before `EnsureRequirements`, which is the default order, so default 
plans don't change. When it doesn't, `EnsureRequirements` resolves any join 
still in `Auto` (this PR), so no join is left in `Auto` in a rule list that 
contains `EnsureRequirements`. Running `JoinSelection` late then costs plan 
quality only: without statistics, a small build side is shuffled instead of 
collected.
   
   This replaces what I wrote on #26481 ("let the planner pick join modes"), 
which you asked about there: once enforcement resolves `Auto`, the physical 
planner doesn't need to. On the PoC branch, with the other rule fixes, the full 
corpus with `JoinSelection` after `EnsureRequirements` has no plan errors or 
wrong results (772 plan errors on main).
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to