Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/10073#discussion_r46355271
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -712,6 +711,52 @@ object PushPredicateThroughAggregate extends
Rule[LogicalPlan] with PredicateHel
}
/**
+ * Reorder the joins so that the bottom ones have at least one condition.
+ */
+object ReorderJoin extends Rule[LogicalPlan] with PredicateHelper {
+
+ /**
+ * Join a list of plans together and push down the conditions into them.
+ *
+ * The joined plan are picked from left to right, prefer those has at
least one join condition.
+ *
+ * @param input a list of LogicalPlans to join.
+ * @param conditions a list of condition for join.
+ */
+ def createOrderedJoin(input: Seq[LogicalPlan], conditions:
Seq[Expression]): LogicalPlan = {
+ assert(input.size >= 2)
+ if (input.size == 2) {
+ Join(input(0), input(1), Inner, conditions.reduceLeftOption(And))
+ } else {
+ val left = input.head
+ val rest = input.drop(1)
+ // find out the first join that have at least one join condition
+ val conditionalJoin = rest.find { plan =>
+ val refs = left.outputSet ++ plan.outputSet
+ conditions.filterNot(_.references.subsetOf(left.outputSet))
+ .filterNot(_.references.subsetOf(plan.outputSet))
+ .exists(cond => cond.references.subsetOf(refs))
+ }
+ // pick the next one if no condition left
+ val right = conditionalJoin.getOrElse(rest.head)
+
+ val joinedRefs = left.outputSet ++ right.outputSet
+ val (joinConditions, others) =
conditions.partition(_.references.subsetOf(joinedRefs))
+ val joined = Join(left, right, Inner,
joinConditions.reduceLeftOption(And))
+
+ createOrderedJoin(Seq(joined) ++ rest.filterNot(_ eq right), others)
--- End diff --
I think that this `eq` is safe, even in the presence of shared subtrees
from self-joins (since the analyzer will rewrite the tree to avoid conflicting
expression ids), but it might be slightly clearer to use `partition` above
instead of `find` if thats not too much work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
