[
https://issues.apache.org/jira/browse/SPARK-7917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617115#comment-14617115
]
Matt Cheah commented on SPARK-7917:
-----------------------------------
[~sowen] was there a patch specifically written in master or 1.4.x that fixed
this? Can you link a specific PR?
> Spark doesn't clean up Application Directories (local dirs)
> ------------------------------------------------------------
>
> Key: SPARK-7917
> URL: https://issues.apache.org/jira/browse/SPARK-7917
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Zach Fry
> Priority: Minor
>
> Similar to SPARK-4834.
> Spark does clean up the cache and lock files in the local dirs, however, it
> doesn't clean up the actual directories.
> We have to write custom scripts to go back through the local dirs and find
> directories that don't contain any files and clear those out.
> Its a pretty simple repro:
> Run a job that does some shuffling, wait for the shuffle files to get cleaned
> up, go and look on disk at spark.local.dir and notice that the directory(s)
> are still there, but there are no files in them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
