Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/12008#discussion_r57641388
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1585,3 +1587,106 @@ object ResolveUpCast extends Rule[LogicalPlan] {
}
}
}
+
+/**
+ * Maps a time column to multiple time windows using the Expand operator.
Since it's non-trivial to
+ * figure out how many windows a time column can map to, we over-estimate
the number of windows and
+ * filter out the rows where the time column is not inside the time window.
+ */
+object TimeWindowing extends Rule[LogicalPlan] {
+
+ /**
+ * Depending on the operation, the TimeWindow expression may be wrapped
in an Alias (in case of
+ * projections) or be simply by itself (in case of groupBy),
+ * @param f The function that we want to apply on the TimeWindow
expression
+ * @return The user defined function applied on the TimeWindow expression
+ */
+ private def getWindowExpr[E](f: TimeWindow => E):
PartialFunction[Expression, E] = {
+ case Alias(windowExpr: TimeWindow, name) =>
+ if (name == windowExpr.toString.toLowerCase) {
--- End diff --
@marmbrus This is a super hacky check for the auto-generated alias in the
project. Do you have any better suggestions?
Also if you can give an example on how it looks like with the extractor, I
can modify it. I didn't want to do too many similar pattern matching
expressions everywhere so I used this function instead
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
