Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/22141#discussion_r211804802
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
---
@@ -137,13 +137,21 @@ object RewritePredicateSubquery extends
Rule[LogicalPlan] with PredicateHelper {
plan: LogicalPlan): (Option[Expression], LogicalPlan) = {
var newPlan = plan
val newExprs = exprs.map { e =>
- e transformUp {
+ e transformDown {
case Exists(sub, conditions, _) =>
val exists = AttributeReference("exists", BooleanType, nullable
= false)()
// Deduplicate conflicting attributes if any.
newPlan = dedupJoin(
Join(newPlan, sub, ExistenceJoin(exists),
conditions.reduceLeftOption(And)))
exists
+ case (Not(InSubquery(values, ListQuery(sub, conditions, _, _)))) =>
+ val exists = AttributeReference("exists", BooleanType, nullable
= false)()
+ val inConditions = values.zip(sub.output).map(EqualTo.tupled)
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
--- End diff --
@liwensun Just to confirm, here is a list of queries i was going to try -
select * from t1 where not (c1 not in (select c2 from t2))
select * from t1 where not (c1 > 5 or c1 in (select c2 from t2))
select * from t1 where not (c1 > 5 or c1 not in (select c2 from t2))
select * from t1 where not (c1 > 5 and c1 in (select c2 from t2))
select * from t1 where not (c1 > 5 and c1 not in (select c2 from t2))
Does this look ok to you ? Please let me know if i missed out on any cases ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
