Repository: spark Updated Branches: refs/heads/master 2a5c93079 -> 6a7ba1ff7
Fix dynamic allocation docs to address cached data. ## What changes were proposed in this pull request? Documentation changes ## How was this patch tested? No tests Author: Michael Gummelt <[email protected]> Closes #12664 from mgummelt/fix-dynamic-docs. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6a7ba1ff Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6a7ba1ff Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6a7ba1ff Branch: refs/heads/master Commit: 6a7ba1ff7431281f4c3994a8db70a8fb6eefbf00 Parents: 2a5c930 Author: Michael Gummelt <[email protected]> Authored: Tue Apr 26 09:31:53 2016 +0100 Committer: Sean Owen <[email protected]> Committed: Tue Apr 26 09:31:53 2016 +0100 ---------------------------------------------------------------------- docs/job-scheduling.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/6a7ba1ff/docs/job-scheduling.md ---------------------------------------------------------------------- diff --git a/docs/job-scheduling.md b/docs/job-scheduling.md index 083c020..40b6cd9 100644 --- a/docs/job-scheduling.md +++ b/docs/job-scheduling.md @@ -158,8 +158,9 @@ executors will fetch shuffle files from the service instead of from each other. shuffle state written by an executor may continue to be served beyond the executor's lifetime. In addition to writing shuffle files, executors also cache data either on disk or in memory. -When an executor is removed, however, all cached data will no longer be accessible. There is -currently not yet a solution for this in Spark 1.2. In future releases, the cached data may be +When an executor is removed, however, all cached data will no longer be accessible. To mitigate this, +by default executors containing cached data are never removed. You can configure this behavior with +`spark.dynamicAllocation.cachedExecutorIdleTimeout`. In future releases, the cached data may be preserved through an off-heap storage similar in spirit to how shuffle files are preserved through the external shuffle service. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
