pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274204571
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +159,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan]
with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan,
rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan,
rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftSemi, joinCond, subHint)
Project(p.output, join)
case (p, Not(Exists(sub, _, _, conditions, subHint))) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan,
rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan,
rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftAnti, joinCond, subHint)
Project(p.output, join)
case (p, InSubquery(values, ListQuery(sub, _, _, _, conditions,
subHint))) =>
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val (joinCond, outerPlan) = rewriteExistentialExpr(inConditions ++
conditions, p)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(inConditions ++ conditions,
p, dedupSub)
Review Comment:
Understood — I will not defer it. Taking it here.
To set expectations on what that adds to this PR, from the measurements in
the thread above: the fix touches both statistics modes, so it changes
estimates for every existing ExistenceJoin, including the ones produced by the
pre-existing outer route that this PR does not touch. Eight approved TPC-DS
plans contain an ExistenceJoin (q10, q35, q45 and their sf100 variants), so
their golden files move too.
I will keep that in a separate commit on this branch — statistics change
plus regenerated goldens — so the subquery-routing fix and the estimation
change stay separately reviewable. Scope as you specified it: left-preserving
row count, the appended Boolean in the output width, left-side-only attribute
statistics, correct behaviour when the right row count is absent or zero,
agreement between the CBO and size-only modes on the left-preserving invariant,
and the build-side plan regression where an otherwise broadcastable left input
stays broadcastable regardless of the right estimate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
