maropu commented on a change in pull request #29304:
URL: https://github.com/apache/spark/pull/29304#discussion_r464149263
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
##########
@@ -391,35 +393,41 @@ object PhysicalWindow {
}
}
-object ExtractSingleColumnNullAwareAntiJoin extends JoinSelectionHelper with
PredicateHelper {
-
- // TODO support multi column NULL-aware anti join in future.
- // See. http://www.vldb.org/pvldb/vol2/vldb09-423.pdf Section 6
- // multi-column null aware anti join is much more complicated than single
column ones.
+object ExtractNullAwareAntiJoinKeys extends JoinSelectionHelper with
PredicateHelper {
// streamedSideKeys, buildSideKeys
private type ReturnType = (Seq[Expression], Seq[Expression])
- /**
- * See. [SPARK-32290]
- * LeftAnti(condition: Or(EqualTo(a=b), IsNull(EqualTo(a=b)))
- * will almost certainly be planned as a Broadcast Nested Loop join,
- * which is very time consuming because it's an O(M*N) calculation.
- * But if it's a single column case O(M*N) calculation could be optimized
into O(M)
- * using hash lookup instead of loop lookup.
- */
def unapply(join: Join): Option[ReturnType] = join match {
- case Join(left, right, LeftAnti,
- Some(Or(e @ EqualTo(leftAttr: AttributeReference, rightAttr:
AttributeReference),
- IsNull(e2 @ EqualTo(_, _)))), _)
- if SQLConf.get.optimizeNullAwareAntiJoin &&
- e.semanticEquals(e2) =>
- if (canEvaluate(leftAttr, left) && canEvaluate(rightAttr, right)) {
- Some(Seq(leftAttr), Seq(rightAttr))
- } else if (canEvaluate(leftAttr, right) && canEvaluate(rightAttr, left))
{
- Some(Seq(rightAttr), Seq(leftAttr))
- } else {
+ case Join(left, right, LeftAnti, condition, _) if
SQLConf.get.optimizeNullAwareAntiJoin =>
+ val predicates = condition.map(splitConjunctivePredicates).getOrElse(Nil)
+ if (predicates.isEmpty ||
+ predicates.length > SQLConf.get.optimizeNullAwareAntiJoinMaxNumKeys) {
None
+ } else {
+ val joinKeys = ArrayBuffer[(Expression, Expression)]()
+
+ // All predicate must match pattern condition: Or(EqualTo(a=b),
IsNull(EqualTo(a=b)))
+ val allMatch = predicates.forall {
+ case Or(e @ EqualTo(leftExpr: Expression, rightExpr: Expression),
+ IsNull(e2 @ EqualTo(_, _))) if e.semanticEquals(e2) =>
Review comment:
IIUC this pattern matching depends on the `RewritePredicateSubquery`
code:
https://github.com/apache/spark/blob/0693d8bbf2942ab96ffe705ef0fc3fe4b0d9ec11/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala#L140
This is okay now, but I'm a little worried that it does not work well if the
`RewritePredicateSubquery` code will be updated; for example, if both
attributes are non-nullable in a join condition, we might be able to remove
`IsNull(c)` for optimization in the `RewritePredicateSubquery` rule.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
