leanken commented on a change in pull request #29304:
URL: https://github.com/apache/spark/pull/29304#discussion_r464154277
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
##########
@@ -391,35 +393,41 @@ object PhysicalWindow {
}
}
-object ExtractSingleColumnNullAwareAntiJoin extends JoinSelectionHelper with
PredicateHelper {
-
- // TODO support multi column NULL-aware anti join in future.
- // See. http://www.vldb.org/pvldb/vol2/vldb09-423.pdf Section 6
- // multi-column null aware anti join is much more complicated than single
column ones.
+object ExtractNullAwareAntiJoinKeys extends JoinSelectionHelper with
PredicateHelper {
// streamedSideKeys, buildSideKeys
private type ReturnType = (Seq[Expression], Seq[Expression])
- /**
- * See. [SPARK-32290]
- * LeftAnti(condition: Or(EqualTo(a=b), IsNull(EqualTo(a=b)))
- * will almost certainly be planned as a Broadcast Nested Loop join,
- * which is very time consuming because it's an O(M*N) calculation.
- * But if it's a single column case O(M*N) calculation could be optimized
into O(M)
- * using hash lookup instead of loop lookup.
- */
def unapply(join: Join): Option[ReturnType] = join match {
- case Join(left, right, LeftAnti,
- Some(Or(e @ EqualTo(leftAttr: AttributeReference, rightAttr:
AttributeReference),
- IsNull(e2 @ EqualTo(_, _)))), _)
- if SQLConf.get.optimizeNullAwareAntiJoin &&
- e.semanticEquals(e2) =>
- if (canEvaluate(leftAttr, left) && canEvaluate(rightAttr, right)) {
- Some(Seq(leftAttr), Seq(rightAttr))
- } else if (canEvaluate(leftAttr, right) && canEvaluate(rightAttr, left))
{
- Some(Seq(rightAttr), Seq(leftAttr))
- } else {
+ case Join(left, right, LeftAnti, condition, _) if
SQLConf.get.optimizeNullAwareAntiJoin =>
+ val predicates = condition.map(splitConjunctivePredicates).getOrElse(Nil)
+ if (predicates.isEmpty ||
+ predicates.length > SQLConf.get.optimizeNullAwareAntiJoinMaxNumKeys) {
None
+ } else {
+ val joinKeys = ArrayBuffer[(Expression, Expression)]()
+
+ // All predicate must match pattern condition: Or(EqualTo(a=b),
IsNull(EqualTo(a=b)))
+ val allMatch = predicates.forall {
+ case Or(e @ EqualTo(leftExpr: Expression, rightExpr: Expression),
+ IsNull(e2 @ EqualTo(_, _))) if e.semanticEquals(e2) =>
Review comment:
yes. the IsNull being removed case is considered, we only do NAAJ
optimize with the Or condition still exists.
Basically, the NAAJ Optimize switch triggered at SparkStrategies, which I
think optimizer is done its job. it's save to put this pattern check in
physical plan state
```
// negative hand-written left anti join
// testData.key nullable false
// testData2.a nullable false
// isnull(key = a) isnull(key+1=a) will be optimized to true literal
and removed
joinExec = assertJoin((
"SELECT * FROM testData LEFT ANTI JOIN testData3 ON (key = a OR
ISNULL(key = a)) " +
"AND (key + 1 = a OR ISNULL(key + 1 = a))",
classOf[BroadcastHashJoinExec]))
assert(!joinExec.asInstanceOf[BroadcastHashJoinExec].isNullAwareAntiJoin)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
