Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9819#discussion_r47465384
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1097,6 +1118,42 @@ class Analyzer(
}
}
}
+
+ /**
+ * Check and add proper window frames for all window functions.
+ */
+ object ResolveWindowFrame extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case logical: LogicalPlan => logical transformExpressions {
+ case WindowExpression(wf: WindowFunction,
+ WindowSpecDefinition(_, _, f: SpecifiedWindowFrame))
+ if wf.frame != UnspecifiedFrame && wf.frame != f =>
+ failAnalysis(s"Window Frame $f must match the required frame
${wf.frame}")
+ case WindowExpression(wf: WindowFunction,
+ s @ WindowSpecDefinition(_, o, UnspecifiedFrame))
+ if wf.frame != UnspecifiedFrame =>
+ WindowExpression(wf, s.copy(frameSpecification = wf.frame))
+ case we @ WindowExpression(e, s @ WindowSpecDefinition(_, o,
UnspecifiedFrame)) =>
+ val frame = SpecifiedWindowFrame.defaultWindowFrame(o.nonEmpty,
acceptWindowFrame = true)
+ we.copy(windowSpec = s.copy(frameSpecification = frame))
+ }
+ }
+ }
+
+ /**
+ * Check and add order to [[AggregateWindowFunction]]s.
+ */
+ object ResolveWindowOrder extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case logical: LogicalPlan => logical transformExpressions {
+ case WindowExpression(wf: WindowFunction, spec) if
spec.orderSpec.isEmpty =>
+ failAnalysis(s"WindowFunction $wf requires window to be ordered")
--- End diff --
All window functions (LEAD, LAG, ROW_NUMBER, RANK, DENSE_RANK,
PERCENT_RANK, NTILE, CUME_DIST) imply ordering. Without ordering their results
would be non-deterministic and not very usefull. Only regular Aggregates make
sense to use in without ordering.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
