Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23057#discussion_r234547323
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
---
@@ -119,7 +139,7 @@ object RewritePredicateSubquery extends
Rule[LogicalPlan] with PredicateHelper {
// (A.A1 = B.B1 OR ISNULL(A.A1 = B.B1)) AND (B.B2 = A.A2) AND
B.B3 > 1
val finalJoinCond = (nullAwareJoinConds ++
conditions).reduceLeft(And)
// Deduplicate conflicting attributes if any.
- dedupJoin(Join(outerPlan, sub, LeftAnti, Option(finalJoinCond)))
+ dedupJoin(Join(outerPlan, newSub, LeftAnti,
Option(finalJoinCond)))
case (p, predicate) =>
val (newCond, inputPlan) =
rewriteExistentialExpr(Seq(predicate), p)
Project(p.output, Filter(newCond.get, inputPlan))
--- End diff --
thanks for your help here @viirya. I added the check also to
`rewriteExistentialExpr`. I was missing the case when it is invoked not only on
the result of `foldLeft`. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
