Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/21330#discussion_r189433323
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1883,7 +1883,19 @@ class Analyzer(
// Second, we group extractedWindowExprBuffer based on their
Partition and Order Specs.
val groupedWindowExpressions = extractedWindowExprBuffer.groupBy {
expr =>
val distinctWindowSpec = expr.collect {
- case window: WindowExpression => window.windowSpec
+ case window: WindowExpression =>
+ val winExpr = window.windowFunction
+ val distinctOpt = winExpr.find (expr =>
expr.isInstanceOf[AggregateExpression]
+ && expr.asInstanceOf[AggregateExpression].isDistinct)
+ if (distinctOpt.nonEmpty &&
window.windowSpec.orderSpec.nonEmpty) {
--- End diff --
So this only works for global window frames? If it does why did you
implement distinct processing for growing, sliding and shrinking frames?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
