pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114428367
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,14 +405,26 @@ object RewritePredicateSubquery extends Rule[LogicalPlan]
with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly
introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite`
returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is
not descended into:
+ * rewriting an existential sub-query nested in its join condition would
graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place
may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression],
LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY,
IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
+ def rewrite(expr: Expression): Expression = {
Review Comment:
Agreed, and fixed. The traversal was replaced in an earlier round to stop it
descending into a
subquery that `canRewrite` declines, and in doing so it lost both things you
list, which
`transformDownWithPruning` provided: the rule was applied inside
`CurrentOrigin.withOrigin(origin)`,
and `copyTagsFrom` ran when a node was replaced.
Each of the three arms now produces its replacement through a `replacing(sq)
{ ... }` helper that
wraps the body in `CurrentOrigin.withOrigin(sq.origin)` and copies the
replaced expression's tags,
so the `exists` attribute, the synthesized join conditions and the `Join`
node are all built under
the origin of the subquery expression they come from. The do-not-descend
behaviour and the pruning
check are unaffected.
The most visible consequence was the `SQLQueryContext` of a runtime error
raised from a rewritten
join condition, which had started pointing at the enclosing operator instead
of the subquery
fragment; that is restored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
