GitHub user joseph-torres opened a pull request:
https://github.com/apache/spark/pull/19611
[SPARK-22305] Write HDFSBackedStateStoreProvider.loadMap non-recursively
## What changes were proposed in this pull request?
Write HDFSBackedStateStoreProvider.loadMap non-recursively. This prevents
stack overflow if too many deltas stack up in a low memory environment.
## How was this patch tested?
existing unit tests for functional equivalence, new unit test to check for
stack overflow
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/joseph-torres/spark SPARK-22305
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19611.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19611
----
commit 6366347516853abd7afd7e89452e656b1011cf6e
Author: Jose Torres <j...@databricks.com>
Date: 2017-10-30T15:48:13Z
rewrite loadMap iteratively
commit 33ea2fb59f5ad47ed4713ca73945a9630486677c
Author: Jose Torres <j...@databricks.com>
Date: 2017-10-30T16:28:28Z
add test exercising stack overflow
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
