[
https://issues.apache.org/jira/browse/SPARK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043102#comment-14043102
]
Sean Owen commented on SPARK-2268:
----------------------------------
Yeah this hook is deleting local files, not HDFS files. I don't think it can
interact with Hadoop APIs or else it fails when used without Hadoop.
> Utils.createTempDir() creates race with HDFS at shutdown
> --------------------------------------------------------
>
> Key: SPARK-2268
> URL: https://issues.apache.org/jira/browse/SPARK-2268
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Marcelo Vanzin
>
> Utils.createTempDir() has this code:
> {code}
> // Add a shutdown hook to delete the temp dir when the JVM exits
> Runtime.getRuntime.addShutdownHook(new Thread("delete Spark temp dir " +
> dir) {
> override def run() {
> // Attempt to delete if some patch which is parent of this is not
> already registered.
> if (! hasRootAsShutdownDeleteDir(dir)) Utils.deleteRecursively(dir)
> }
> })
> {code}
> This creates a race with the shutdown hooks registered by HDFS, since the
> order of execution is undefined; if the HDFS hooks run first, you'll get
> exceptions about the file system being closed.
> Instead, this should use Hadoop's ShutdownHookManager with a proper priority,
> so that it runs before the HDFS hooks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
