GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/22583
[SPARK-10816][SS] SessionWindow support for Structure Streaming
## What changes were proposed in this pull request?
As the discussion in
[SPARK-10816](https://issues.apache.org/jira/browse/SPARK-10816), here we add
the support for session window support in Structure Streaming. For convenient
review, this PR can be split to several part:
- Implement for session_window build-in function in dataset api, add
related analysis rule and exec node.
- Implement for integrating with streaming frame work, including the
SessionWindowStateStoreRestoreExec/SessionWindowMergeExec/SessionWindowStateStoreSaveExec.
## How was this patch tested?
Test for dataset\sql only and end-to-end streaming tests.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuanyuanking/spark SPARK-10816
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22583.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22583
----
commit 5e4034ff26f3bab65d44a142aa29ec5b2082e81f
Author: liyuanjian <liyuanjian@...>
Date: 2018-08-02T09:38:04Z
INF-SPARK-1474: add build.sh and BCLOUD for new branch
commit 2d943d5a4e7659a22686ac5490d8aecd63d939ba
Author: liyuanjian <liyuanjian@...>
Date: 2018-08-02T10:16:22Z
acu-bsc-15: add session window expression
commit 55b8196e6622710dfed3b25305a3ec408af772fb
Author: liyuanjian <liyuanjian@...>
Date: 2018-08-16T09:01:33Z
acu-bsc-19: Add support in logical\pysical plan for session window, add UT
in DataFrameSessionWindowingSuite
commit 05dcd7f3e5572ff429632407099afa9d536e36e3
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-09-28T06:47:56Z
Fix for spark master baned transform in Analyzer
commit b138f09479523d6386137c4c6a0b7dbe03b97273
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-08-22T13:18:43Z
Addression comments and code clean
commit a2517b610cbb38123d1edf4eda2a5186d44b1a22
Author: liyuanjian <liyuanjian@...>
Date: 2018-08-23T06:44:15Z
Address comment and bug fix
commit 29567883fb76911b0bc3614cf5b0c910d6a0b200
Author: huangtengfei02 <huangtengfei02@...>
Date: 2018-08-29T09:11:22Z
acu-bsc-40: add SessionWindowMergeExec
Change-Id: Ie9669ee7827f0e52d2837ddd8a840a07543850ba
commit cb770c76147bbc20bb3627c913c3526b2f8dc161
Author: zhuliangchang <zhuliangchang@...>
Date: 2018-09-05T04:10:33Z
acu-bsc-20, StateStore management of SessionWindow
1. acu-bsc-21, SessionWindowStateStoreRestore by huangtengfei02
<[email protected]>
2. acu-bsc-22, SessionWindowStateStoreSave of complete / append mode
3. acu-bsc-18, new Strategy for Streaming query with SessionWindow
4. acu-bsc-17, generate new aggregate physical plan in AggUtils
commit d65015e970a126e6f128aab63a47f436f272028c
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-09-28T09:42:51Z
Remove internal build script
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
