Github user aokolnychyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/19193#discussion_r156495899
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1920,7 +1927,34 @@ class Analyzer(
case p: LogicalPlan if !p.childrenResolved => p
- // Aggregate without Having clause.
+ // Extract window expressions from aggregate functions. There might
be an aggregate whose
+ // aggregate function contains a window expression as a child, which
we need to extract.
+ // e.g., df.groupBy().agg(max(rank().over(window))
+ case a @ Aggregate(groupingExprs, aggregateExprs, child)
+ if containsAggregateFunctionWithWindowExpression(aggregateExprs) &&
+ a.expressions.forall(_.resolved) =>
+
+ val windowExprAliases = new ArrayBuffer[NamedExpression]()
+ val newAggregateExprs = aggregateExprs.map { expr =>
+ expr.transform {
--- End diff --
Hmm, do you actually mean smth like this?
```
val df = Seq((1, 2), (1, 3), (2, 4), (5, 5)).toDF("a", "b")
val window1 = Window.orderBy('a)
df.groupBy('a).agg(max(sum(sum('b)).over(window1))).explain(true)
df.groupBy('a).agg(max(sum(sum('b)).over(window1))).show(false)
```
```
== Analyzed Logical Plan ==
a: int, max(sum(sum(b)) OVER (ORDER BY a ASC NULLS FIRST
unspecifiedframe$())): bigint
Aggregate [a#5], [a#5, max(_we0#22L) AS max(sum(sum(b)) OVER (ORDER BY a
ASC NULLS FIRST unspecifiedframe$()))#16L]
+- Project [a#5, b#6, _we0#22L, _we0#22L]
+- Window [sum(sum(cast(b#6 as bigint))) windowspecdefinition(a#5 ASC
NULLS FIRST, specifiedwindowframe(RangeFrame, unboundedpreceding$(),
currentrow$())) AS _we0#22L], [a#5 ASC NULLS FIRST]
+- Project [_1#2 AS a#5, _2#3 AS b#6]
+- LocalRelation [_1#2, _2#3]
== Optimized Logical Plan ==
Aggregate [a#5], [a#5, max(_we0#22L) AS max(sum(sum(b)) OVER (ORDER BY a
ASC NULLS FIRST unspecifiedframe$()))#16L]
+- Project [a#5, _we0#22L, _we0#22L]
+- Window [sum(sum(cast(b#6 as bigint))) windowspecdefinition(a#5 ASC
NULLS FIRST, specifiedwindowframe(RangeFrame, unboundedpreceding$(),
currentrow$())) AS _we0#22L], [a#5 ASC NULLS FIRST]
+- LocalRelation [a#5, b#6]
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
