[
https://issues.apache.org/jira/browse/SPARK-7917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617166#comment-14617166
]
Matt Cheah edited comment on SPARK-7917 at 7/7/15 6:45 PM:
-----------------------------------------------------------
Definitely not SPARK-7503 - the PR there only did things for YARN mode:
https://github.com/apache/spark/pull/6026
was (Author: mcheah):
Definitely not 7503 - the PR there only did things for YARN mode:
https://github.com/apache/spark/pull/6026
> Spark doesn't clean up Application Directories (local dirs)
> ------------------------------------------------------------
>
> Key: SPARK-7917
> URL: https://issues.apache.org/jira/browse/SPARK-7917
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Zach Fry
> Priority: Minor
>
> Similar to SPARK-4834.
> Spark does clean up the cache and lock files in the local dirs, however, it
> doesn't clean up the actual directories.
> We have to write custom scripts to go back through the local dirs and find
> directories that don't contain any files and clear those out.
> Its a pretty simple repro:
> Run a job that does some shuffling, wait for the shuffle files to get cleaned
> up, go and look on disk at spark.local.dir and notice that the directory(s)
> are still there, but there are no files in them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
