cloud-fan commented on code in PR #58269:
URL: https://github.com/apache/spark/pull/58269#discussion_r3935971993
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InlineCTE.scala:
##########
@@ -89,29 +89,55 @@ case class InlineCTE(
}
private def validateNoOuterReferencesAcrossCTEBoundary(cteDef:
CTERelationDef): Unit = {
- val outerRefs = cteDef.child.flatMap(
- _.expressions.flatMap(_.collect { case o: OuterReference => o }))
- if (outerRefs.nonEmpty) {
+ // Only an outer reference that actually escapes the CTE definition is
invalid. An outer
+ // reference that points to an attribute produced by an operator inside
the definition
+ // body (e.g. a correlated subquery whose correlated column lives in the
body) does not
+ // escape, so the definition is self-contained and safe to materialize.
Walk the whole
+ // definition (main tree plus nested subquery plans) once, collect every
attribute bound
+ // anywhere in the definition, and reject only references that resolve to
none of them.
+ //
+ // Matching by exprId is exact for plans produced by a single analysis
pass: every
+ // `OuterReference` wraps the very attribute it binds to, exprIds are
allocated fresh
+ // per resolution, and `DeduplicateRelations` rewrites duplicated relation
instances
+ // (self-join, union, etc.) with fresh exprIds before the optimizer runs.
The known
+ // residual gap is a plan stitched together from already-analyzed plans
(e.g. a producer
+ // embedding the same analyzed dataset twice): such a tree can carry the
same exprId in
+ // multiple places, so an escaping reference could match a duplicated
exprId and slip
+ // through. Such a query is ill-formed and fails later during planning or
execution
+ // anyway; this check is defense-in-depth that fails fast with a clear
error otherwise.
+ val allNodes = cteDef.child.collectWithSubqueries { case n: LogicalPlan =>
n }
+ val boundExprIds = allNodes.iterator
+ .flatMap(_.output.filter(_.resolved).map(_.exprId))
Review Comment:
Thanks for tracing the analyzer contract here. I agree that both masking
shapes require bypassing analysis: projected outer references are aliased,
conflicting embedded plans are deduplicated before optimization, and the
residual stitched-plan limitation is documented. This resolves my concern.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"inline:3933711073","thread_id":"inline:3933711073","verdict_sha256":"e5c35005451a40f4ace86649d78bab420032ce4855b51821ff2f8bc0cfb5dbf2"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
