[
https://issues.apache.org/jira/browse/SPARK-7917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617177#comment-14617177
]
Sean Owen commented on SPARK-7917:
----------------------------------
Right, this is about standalone. There's
https://github.com/apache/spark/pull/3705 but that was in 1.3. IIRC it looks
like this dir gets cleaned up pretty reliably on worker shutdown if the JVM can
exit pretty normally, so I think the question is, does it still happen on
master? and wha causes the normal code path not to happen?
> Spark doesn't clean up Application Directories (local dirs)
> ------------------------------------------------------------
>
> Key: SPARK-7917
> URL: https://issues.apache.org/jira/browse/SPARK-7917
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Zach Fry
> Priority: Minor
>
> Similar to SPARK-4834.
> Spark does clean up the cache and lock files in the local dirs, however, it
> doesn't clean up the actual directories.
> We have to write custom scripts to go back through the local dirs and find
> directories that don't contain any files and clear those out.
> Its a pretty simple repro:
> Run a job that does some shuffling, wait for the shuffle files to get cleaned
> up, go and look on disk at spark.local.dir and notice that the directory(s)
> are still there, but there are no files in them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
