[
https://issues.apache.org/jira/browse/SPARK-7439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14537560#comment-14537560
]
Taeyun Kim commented on SPARK-7439:
-----------------------------------
What's registered are the subdirectories of the 'temp' directory.
The 'temp' directories themselves are not registered.
See DiskBlockManager.createLocalDirs(). It throws away the 'rootDir' variable
which contains the information for the newly created and not marked unique
'temp' directory.
The 'temp' directories remain whether the app exits normally or not.
> Should delete temporary local directories
> -----------------------------------------
>
> Key: SPARK-7439
> URL: https://issues.apache.org/jira/browse/SPARK-7439
> Project: Spark
> Issue Type: Bug
> Components: Block Manager
> Affects Versions: 1.3.1
> Environment: Windows 7, CentOS 6.6
> Reporter: Taeyun Kim
> Priority: Minor
>
> Spark does not delete temporary local directories.
> After a spark program completes, there are 3 temporary directories remain in
> the temp directory. The directory names are like this:
> spark-2e389487-40cc-4a82-a5c7-353c0feefbb7
> The directories are empty.
> They are created every time the Spark program runs. So the number of files
> and directories keeps growing.
> I've traced the spark source code.
> The module methods that create the 3 'temp' directories are as follows:
> * DiskBlockManager.createLocalDirs
> * HttpFileServer.initialize
> * SparkEnv.sparkFilesDir
> They (eventually) call Utils.getOrCreateLocalRootDirs and then
> Utils.createDirectory, which intentionally does NOT mark the directory for
> automatic deletion.
> The comment of createDirectory method says: "The directory is guaranteed to
> be newly created, and is not marked for automatic deletion."
> But since the directories does not hold useful data after the program
> completes, they should be deleted if possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
