GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/6482
[SPARK-7930][CORE][STREAMING] Fixed shutdown hook priorities
Shutdown hook for temp directories had priority 100 while SparkContext was
50. So the local root directory was deleted before SparkContext was shutdown.
This leads to scary errors on running jobs, at the time of shutdown. This is
especially a problem when running streaming examples, where Ctrl-C is the only
way to shutdown.
The fix in this PR is to make the temp directory shutdown priority lower
than SparkContext, so that the temp dirs are the last thing to get deleted,
after the SparkContext has been shut down. Also, the DiskBlockManager shutdown
priority is change from default 100 to temp_dir_prio + 1, so that it gets
invoked just before all temp dirs are cleared.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-7930
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6482.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6482
----
commit 1514d0b2e10be478cb48978b480e4ecc2e9b9843
Author: Tathagata Das <[email protected]>
Date: 2015-05-29T00:58:02Z
Fixed shutdown hook priorities
commit d7cbeb5d3a657ca0e6596d67a022561ff48835d2
Author: Tathagata Das <[email protected]>
Date: 2015-05-29T01:02:49Z
Removed unnecessary line
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
