GitHub user aalobaidi opened a pull request:
https://github.com/apache/spark/pull/21500
Scalable Memory option for HDFSBackedStateStore
More scalable memory management for HDFSBackedStateStore. This is
controlled by a configuration
(`spark.sql.streaming.stateStore.unloadAfterCommit`), if enabled
HDFSBackedStateStore will unload state after commit.
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
This is been tested manually but need unit tests.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aalobaidi/spark Memory-HDFSBackedStateStore
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21500.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21500
----
commit fecbc23ec47c30d69d58bcfa3573751b5432eba9
Author: Ahmed Al-Obaidi <ahmeda@...>
Date: 2018-03-31T19:43:45Z
Scalable Memory option for HDFSBackedStateStore
Allow configuration option to unload loadedMaps from memory after commit
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
