GitHub user brkyvz opened a pull request:
https://github.com/apache/spark/pull/12136
[SPARK-14353] Dateset Time Window `window` API for Python, SQL, and R
## What changes were proposed in this pull request?
The `window` function was added to Dataset with [this
PR](https://github.com/apache/spark/pull/12008).
This PR adds the Python, SQL, and R API for this function.
With this PR, SQL, Java, and Scala will share the same APIs as in users can
use:
- `window(timeColumn, windowDuration)`
- `window(timeColumn, windowDuration, slideDuration)`
- `window(timeColumn, windowDuration, slideDuration, startTime)`
In Python and R, users can access all APIs above, but in addition they can
do
- In Python:
`window(timeColumn, windowDuration, startTime=...)`
- In R:
`window(df$timeColumn, windowDuration, startTime=...)`
that is, they can provide the startTime without providing the
`slideDuration`. In this case, we will generate tumbling windows.
## How was this patch tested?
Unit tests + manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/brkyvz/spark python-windows
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12136.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12136
----
commit 4777b774b0aba00aeaf30d30eed344cd0e37df6e
Author: Burak Yavuz <[email protected]>
Date: 2016-04-03T03:39:42Z
added python api for time windowing
commit 7b0fb13e1553a765c371f6eb38bda7e23c846eb3
Author: Burak Yavuz <[email protected]>
Date: 2016-04-03T07:05:12Z
everything except R
commit f589469b9611b4560cb4eac262bb4f554f6c2bb7
Author: Burak Yavuz <[email protected]>
Date: 2016-04-03T08:31:41Z
added window support for sql, python, and R
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
