pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4072311118
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +158,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan]
with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan,
rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
Review Comment:
Accepted in full — this was a regression I introduced when I replaced the
traversal, not a
pre-existing gap, and your reading of the two lines is exactly right.
Each arm now builds its replacement through a small helper,
```scala
def replacing(sq: Expression)(replacement: => Expression): Expression = {
val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
newExpr.copyTagsFrom(sq)
newExpr
}
```
so everything an arm constructs — the `exists` attribute, the join
conditions and the `Join` itself
— is built under the origin of the subquery expression being replaced, and
that expression's tags
are carried over, which is what `transformDownWithPruning` did.
Your note about where the cost lands is what decided the shape of the fix:
`handleUnaryNode` sends
every `UnaryNode` through this method with the default `canRewrite`, so the
repair had to sit in the
arms themselves rather than in the new routing callers. The pruning check
and the unchanged-node
fast path are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
