pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4286934055
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,12 @@ object RewritePredicateSubquery extends Rule[LogicalPlan]
with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential
sub-queries that only the
+ // sub-query plan can evaluate, see
rewriteExistentialExprInJoinCondition.
+ val (subConditions, newSub) =
+ rewriteExistentialExprInSubqueryPlan(conditions, p, dedupSub)
Review Comment:
Confirmed and fixed. I verified the regression against the true merge base
this time:
| predicate (outer `NOT IN`) | master `6e5c61fb` | before this commit | now
| DuckDB |
|---|---|---|---|---|
| `(c1 IN (SELECT col1 FROM t3n)) <=> false` | `2, 3` | `0A000` | `2, 3` |
`2, 3` |
| `(c1 IN (...)) IS NULL` | `2, 3` | `0A000` | `2, 3` | `2, 3` |
| `COALESCE(c1 IN (...), true)` | `2, 3` | `0A000` | `2, 3` | `2, 3` |
| `(c1 NOT IN (...)) <=> false` | `2, 3` | `0A000` | `2, 3` | `2, 3` |
`rewriteExistentialExprInSubqueryPlan` takes a `keepsThreeValuedResiduals`
flag, passed only by the NOT IN arm. Under it an uncorrelated,
unknown-sensitive, subquery-plan-owned nested IN is partitioned out of the
rejection and excluded from `canRewrite`, so it stays in the anti-join
condition exactly as at the merge base. Correlated ones -- including ones that
only carry a correlated descendant -- are still rejected, and truth-only ones
are still rewritten.
As you asked, the exemption is tied to the bare-`Join` result in a comment
at the call site, so a future change that wraps it and brings the
join-condition handling into reach does not silently start two-valuing these
residuals.
New test `NOT IN keeps an uncorrelated three-valued residual` covers all
four shapes, plus a correlated nested IN under both an outer `NOT IN` and an
outer `EXISTS` still being rejected, and a truth-only nested IN under `NOT IN`
still being rewritten. With the exemption removed it fails on the first shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
