c21 commented on a change in pull request #31567:
URL: https://github.com/apache/spark/pull/31567#discussion_r577339415
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -539,17 +539,22 @@ object LimitPushDown extends Rule[LogicalPlan] {
// pushdown Limit.
case LocalLimit(exp, u: Union) =>
LocalLimit(exp, u.copy(children =
u.children.map(maybePushLocalLimit(exp, _))))
- // Add extra limits below OUTER JOIN. For LEFT OUTER and RIGHT OUTER JOIN
we push limits to
- // the left and right sides, respectively. It's not safe to push limits
below FULL OUTER
- // JOIN in the general case without a more invasive rewrite.
+ // Add extra limits below JOIN. For LEFT OUTER and RIGHT OUTER JOIN we
push limits to
+ // the left and right sides, respectively. For INNER and CROSS JOIN we
push limits to
+ // both the left and right sides if join condition is empty. It's not safe
to push limits
+ // below FULL OUTER JOIN in the general case without a more invasive
rewrite.
// We also need to ensure that this limit pushdown rule will not
eventually introduce limits
// on both sides if it is applied multiple times. Therefore:
// - If one side is already limited, stack another limit on top if the
new limit is smaller.
// The redundant limit will be collapsed by the CombineLimits rule.
- case LocalLimit(exp, join @ Join(left, right, joinType, _, _)) =>
+ case LocalLimit(exp, join @ Join(left, right, joinType, conditionOpt, _))
=>
val newJoin = joinType match {
case RightOuter => join.copy(right = maybePushLocalLimit(exp, right))
case LeftOuter => join.copy(left = maybePushLocalLimit(exp, left))
Review comment:
It seems that we can also push down limit into left side, for LEFT SEMI
and LEFT ANTI join, right?
I can create a minor PR if it's not on your plan @wangyum , thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
