GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/11645
[WIP][SPARK-13809][SQL] State store for streaming aggregations
## What changes were proposed in this pull request?
In this PR, I am implementing a new abstraction for management of streaming
state data - State Store. It is a key-value store for persisting running
aggregates for aggregate operations in streaming dataframes. The motivation and
design is discussed here.
https://docs.google.com/document/d/1-ncawFx8JS5Zyfq1HAEGBx56RDet9wfVp_hDM8ZL254/edit#
## How was this patch tested?
Unit tests (WIP)
## TODO
[ ] Unit tests
- [ ] StateStoreCoordinator
- [ ] StateStoreRDD fault-tolerance
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark state-store
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11645.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11645
----
commit c0194744378c2b1827aa6bb3a6109b3549710a44
Author: Tathagata Das <[email protected]>
Date: 2016-03-10T23:02:37Z
First draft of StateStore
commit 4f8dade98ab2720ffab78a6ace567b9705f1b488
Author: Tathagata Das <[email protected]>
Date: 2016-03-11T01:50:50Z
Updated tests
commit f417bde8bd65572e78125c7aa900221fa0bc4d7c
Author: Tathagata Das <[email protected]>
Date: 2016-03-11T04:24:39Z
Added basic unit test for StateStoreRDD
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
