GitHub user aalobaidi opened a pull request: https://github.com/apache/spark/pull/21500
Scalable Memory option for HDFSBackedStateStore More scalable memory management for HDFSBackedStateStore. This is controlled by a configuration (`spark.sql.streaming.stateStore.unloadAfterCommit`), if enabled HDFSBackedStateStore will unload state after commit. ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? This is been tested manually but need unit tests. Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aalobaidi/spark Memory-HDFSBackedStateStore Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21500.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21500 ---- commit fecbc23ec47c30d69d58bcfa3573751b5432eba9 Author: Ahmed Al-Obaidi <ahmeda@...> Date: 2018-03-31T19:43:45Z Scalable Memory option for HDFSBackedStateStore Allow configuration option to unload loadedMaps from memory after commit ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org