Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14057
This breaks null aware anti joins in most cases.
For example:
`select * from a where a.id not in (select id from b)`
Get planned as:
```
== Parsed Logical Plan ==
'Project [*]
+- 'Filter NOT 'a.id IN (list#16)
: +- 'SubqueryAlias list#16
: +- 'Project ['id]
: +- 'UnresolvedRelation `b`
+- 'UnresolvedRelation `a`
== Analyzed Logical Plan ==
id: bigint
Project [id#0L]
+- Filter NOT predicate-subquery#16 [(id#0L = id#6L)]
: +- SubqueryAlias predicate-subquery#16 [(id#0L = id#6L)]
: +- Project [id#6L]
: +- SubqueryAlias b
: +- Range (5, 10, splits=8)
+- SubqueryAlias a
+- Range (0, 10, splits=8)
== Optimized Logical Plan ==
Join LeftAnti, (isnull((id#0L = id#6L)) || (id#0L = id#6L))
:- Range (0, 10, splits=8)
+- Range (5, 10, splits=8)
== Physical Plan ==
BroadcastNestedLoopJoin BuildRight, LeftAnti, (isnull((id#0L = id#6L)) ||
(id#0L = id#6L)), true
:- *Range (0, 10, splits=8)
+- BroadcastExchange IdentityBroadcastMode
+- *Range (5, 10, splits=8)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
