[
https://issues.apache.org/jira/browse/SPARK-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571599#comment-15571599
]
Sean Owen commented on SPARK-17893:
-----------------------------------
OK, well windows here aren't inherently including "past" or "future" events;
they are just a set of events spanning a certain amount of time, and aren't
relative to the time of a particular event in the window. You can compute
aggregates over the window as with other grouping functions. It sounds like you
just want to use "7 days" as the window size and slide duration of "1 day".
Each row of the resulting aggregation represents the 7 days leading up to a
different unique day as desired.
As a side note, "1 day" here means 86,400,000 ms and not a calendar day. If you
really need calendar days, this is probably trickier to get exactly right. You
would probably have to first aggregate by day, and then window over 7 rows
preceding to truly span 7 days each time. That assumes that there is some data
for every day though.
> Window functions should also allow looking back in time
> -------------------------------------------------------
>
> Key: SPARK-17893
> URL: https://issues.apache.org/jira/browse/SPARK-17893
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 2.0.1
> Reporter: Raviteja Lokineni
>
> This function should allow looking back. The current window(timestamp,
> duration) seems to be for looking forward in time.
> Example:
> {code}dataFrame.groupBy(window("date", "7 days ago")).agg(min("col1"),
> max("col1")){code}
> For example, if date: 2013-01-07 then the window should be 2013-01-01 -
> 2013-01-07
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
