peter-toth commented on code in PR #40137:
URL: https://github.com/apache/spark/pull/40137#discussion_r1115359045
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/AliasAwareOutputExpression.scala:
##########
@@ -114,7 +114,11 @@ trait AliasAwareQueryOutputOrdering[T <: QueryPlan[T]]
}
}.takeWhile(_.isDefined).flatten.toSeq
} else {
- orderingExpressions
+ // Make sure the returned ordering are valid (only reference output
attributes of the current
Review Comment:
Good catch! Thanks for this follow-up!
But, I'm not sure this part is correct as `sortOrder.children`s should be
filtered with `_.references.subsetOf(outputSet)` separately.
E.g. this test (is a bit artifical though) fails now:
```
test("SPARK-42049: Improve AliasAwareOutputExpression - no alias but still
prune expressions 2") {
withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
Seq(CollapseProject.ruleName, ColumnPruning.ruleName).mkString(",")) {
val df = spark.range(2).select($"id" as "a", $"id" as "b").select($"a")
val outputOrdering = df.queryExecution.optimizedPlan.outputOrdering
assert(outputOrdering.size == 1)
assert(outputOrdering.head.child.asInstanceOf[Attribute].name == "a")
assert(outputOrdering.head.sameOrderExpressions.size == 0)
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
