Github user aokolnychyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19193#discussion_r156495899 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1920,7 +1927,34 @@ class Analyzer( case p: LogicalPlan if !p.childrenResolved => p - // Aggregate without Having clause. + // Extract window expressions from aggregate functions. There might be an aggregate whose + // aggregate function contains a window expression as a child, which we need to extract. + // e.g., df.groupBy().agg(max(rank().over(window)) + case a @ Aggregate(groupingExprs, aggregateExprs, child) + if containsAggregateFunctionWithWindowExpression(aggregateExprs) && + a.expressions.forall(_.resolved) => + + val windowExprAliases = new ArrayBuffer[NamedExpression]() + val newAggregateExprs = aggregateExprs.map { expr => + expr.transform { --- End diff -- Hmm, do you actually mean smth like this? ``` val df = Seq((1, 2), (1, 3), (2, 4), (5, 5)).toDF("a", "b") val window1 = Window.orderBy('a) df.groupBy('a).agg(max(sum(sum('b)).over(window1))).explain(true) df.groupBy('a).agg(max(sum(sum('b)).over(window1))).show(false) ``` ``` == Analyzed Logical Plan == a: int, max(sum(sum(b)) OVER (ORDER BY a ASC NULLS FIRST unspecifiedframe$())): bigint Aggregate [a#5], [a#5, max(_we0#22L) AS max(sum(sum(b)) OVER (ORDER BY a ASC NULLS FIRST unspecifiedframe$()))#16L] +- Project [a#5, b#6, _we0#22L, _we0#22L] +- Window [sum(sum(cast(b#6 as bigint))) windowspecdefinition(a#5 ASC NULLS FIRST, specifiedwindowframe(RangeFrame, unboundedpreceding$(), currentrow$())) AS _we0#22L], [a#5 ASC NULLS FIRST] +- Project [_1#2 AS a#5, _2#3 AS b#6] +- LocalRelation [_1#2, _2#3] == Optimized Logical Plan == Aggregate [a#5], [a#5, max(_we0#22L) AS max(sum(sum(b)) OVER (ORDER BY a ASC NULLS FIRST unspecifiedframe$()))#16L] +- Project [a#5, _we0#22L, _we0#22L] +- Window [sum(sum(cast(b#6 as bigint))) windowspecdefinition(a#5 ASC NULLS FIRST, specifiedwindowframe(RangeFrame, unboundedpreceding$(), currentrow$())) AS _we0#22L], [a#5 ASC NULLS FIRST] +- LocalRelation [a#5, b#6] ```
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org