Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/10678#discussion_r50791831
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -521,38 +522,96 @@ class Analyzer(
*/
object ResolveSortReferences extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
- case s @ Sort(ordering, global, p @ Project(projectList, child))
- if !s.resolved && p.resolved =>
- val (newOrdering, missing) = resolveAndFindMissing(ordering, p,
child)
+ // Here, this rule only resolves the missing sort references if the
child is not Aggregate
+ // Another rule ResolveAggregateFunctions will resolve that case.
+ case s @ Sort(_, _, child)
+ if !s.resolved && child.resolved &&
!child.isInstanceOf[Aggregate] =>
+ val (newOrdering, missingResolvableAttrs) =
collectResolvableMissingAttrs(s.order, child)
+
+ if (missingResolvableAttrs.isEmpty) {
+ val unresolvableAttrs = s.order.filterNot(_.resolved)
+ logDebug(s"Failed to find $unresolvableAttrs in
${child.output.mkString(", ")}")
+ s // Nothing we can do here. Return original plan.
+ } else {
+ // Add the missing attributes into projectList of Project/Window
or
+ // aggregateExpressions of Aggregate, if they are in the
inputSet
+ // but not in the outputSet of the plan.
+ val newChild = child transformUp {
+ case p: Project =>
+ p.copy(projectList = p.projectList ++
+ missingResolvableAttrs.filter((p.inputSet --
p.outputSet).contains))
+ case w: Window =>
+ w.copy(projectList = w.projectList ++
+ missingResolvableAttrs.filter((w.inputSet --
w.outputSet).contains))
+ case a: Aggregate =>
--- End diff --
In the following query, we can trigger this case. Actually, this query is
based on the failed TPCDS queries.
```select area, rank() over (partition by area order by month) as c1 from
windowData group by product, area, month order by product, area```
If we remove this case, we will get this error:
```
Failed to analyze query: org.apache.spark.sql.AnalysisException: resolved
attribute(s) product#2 missing from area#1,c1#39 in operator !Sort [product#2
ASC,area#1 ASC], true;
Project [area#1,c1#48]
+- !Sort [product#2 ASC,area#1 ASC], true
+- Project [area#1,c1#48]
+- Project [area#1,month#0,c1#48,c1#48]
+- Window [area#1,month#0], [rank(month#0)
windowspecdefinition(area#1,month#0 ASC,ROWS BETWEEN UNBOUNDED PRECEDING AND
CURRENT ROW) AS c1#48], [area#1], [month#0 ASC]
+- Aggregate [product#2,area#1,month#0], [area#1,month#0]
+- Subquery windowdata
+- LogicalRDD [month#0,area#1,product#2],
MapPartitionsRDD[1] at apply at Transformer.scala:22
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
