EnricoMi commented on code in PR #39754:
URL: https://github.com/apache/spark/pull/39754#discussion_r1454788348
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -3286,6 +3265,25 @@ class Analyzer(override val catalogManager:
CatalogManager) extends RuleExecutor
}
}
+ /**
+ * Resolves expressions against their scope of attributes.
+ */
+ object ResolveScopedExpression extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
plan.resolveExpressions {
+ case se: ScopedExpression if se.resolved => se.child
+ case se @ ScopedExpression(expr, attributes) =>
Review Comment:
> apply special logic if it's AppendColumns,
Sounds like you still want to bypass the children if it is a
`AppendColumns`, as we used to do in (and few other places):
```
object ResolveReferencesInAggregate extends SQLConfHelper
with ColumnResolutionHelper with AliasHelper {
def apply(a: Aggregate): Aggregate = {
val planForResolve = a.child match {
// SPARK-25942: Resolves aggregate expressions with `AppendColumns`'s
children, instead of
// `AppendColumns`, because `AppendColumns`'s serializer might produce
conflict attribute
// names leading to ambiguous references exception.
case appendColumns: AppendColumns => appendColumns
case _ => a
}
...
```
We cannot know whether the child of a `MapGroups` / `CoGroup` is an
`AppendColumns` that adds the key column (which we want to bypass or ignore),
or an `AppendColumns` that comes from `mapValues` (which we do not want to
ignore). See the _"However, mapValues introduces another 'value' column, ..."_
section in the PR description above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
