[
https://issues.apache.org/jira/browse/SPARK-7917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617278#comment-14617278
]
Sean Owen commented on SPARK-7917:
----------------------------------
Oops I mean executor. At least, I'm looking at Utils.getLocalFile, which
ultimately calls getOrCreateLocalRootDirsImpl. You can see that on YARN, this
uses YARN's dir and doesn't delete it on exit (YARN manages it). In the case
that spark.local.dir config takes hold, you can also see it creates the dir if
it doesn't exist and will delete it on shutdown in that case.
However I suppose a few possible cases jump out where the dir is not deleted:
- SPARK_EXECUTOR_DIRS is set
- spark.local.dir is set but it already exists
That is it seems to not delete dirs that were managed or set up externally.
Does that explain this maybe?
> Spark doesn't clean up Application Directories (local dirs)
> ------------------------------------------------------------
>
> Key: SPARK-7917
> URL: https://issues.apache.org/jira/browse/SPARK-7917
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Zach Fry
> Priority: Minor
>
> Similar to SPARK-4834.
> Spark does clean up the cache and lock files in the local dirs, however, it
> doesn't clean up the actual directories.
> We have to write custom scripts to go back through the local dirs and find
> directories that don't contain any files and clear those out.
> Its a pretty simple repro:
> Run a job that does some shuffling, wait for the shuffle files to get cleaned
> up, go and look on disk at spark.local.dir and notice that the directory(s)
> are still there, but there are no files in them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
