GitHub user velvia opened a pull request:
https://github.com/apache/spark/pull/288
SPARK-1154: Clean up app folders in worker nodes
This is a fix for
[SPARK-1154](https://issues.apache.org/jira/browse/SPARK-1154). The issue is
that worker nodes fill up with a huge number of app-* folders after some time.
This change adds a periodic cleanup task which asynchronously deletes app
directories older than a configurable TTL.
Two new configuration parameters have been introduced:
spark.worker.cleanup_interval
spark.worker.app_data_ttl
This change does not include moving the downloads of application jars to a
location outside of the work directory. We will address that if we have time,
but that potentially involves caching so it will come either as part of this PR
or a separate PR.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/velvia/spark SPARK-1154-cleanup-app-folders
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/288.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #288
----
commit 6a9eed14e28af301966566c7e2566cca7808cfd9
Author: Evan Chan <[email protected]>
Date: 2014-04-01T18:04:33Z
SPARK-1154: Add a periodic task to clean up app directories
This adds two config params:
spark.worker.cleanup_interval
spark.worker.app_data_ttl
commit 574bc82743a5a8fcf3c0b3185c1c9166c54b8655
Author: Evan Chan <[email protected]>
Date: 2014-04-01T18:31:29Z
Document the two new settings
commit 31961673a4f3c3474e8bb1594153d4908e7f8d11
Author: Evan Chan <[email protected]>
Date: 2014-04-01T20:30:54Z
Don't recompute current time with every new file
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
