GitHub user BryanCutler opened a pull request:
https://github.com/apache/spark/pull/11746
[SPARK-13602][CORE] Add shutdown hook to DriverRunner to prevent driver
process leak
## What changes were proposed in this pull request?
Added shutdown hook to DriverRunner to kill the driver process in case the
Worker JVM exits suddenly and the `WorkerWatcher` was unable to properly catch
this. Changed the `DriverRunner.kill()` method to make use of a Thread
interrupt instead of setting a flag so that the process can be managed within
the Thread scope. Did some cleanup to consolidate driver state management and
setting of finalized vars within the running thread.
## How was this patch tested?
Added unit tests toverify that final state and exception variables are set
accordingly for successfull, failed, and errors in the driver process.
Retrofitted existing test to verify killing of mocked process ends with the
correct state and stops properly
Manually tested (with deploy-mode=cluster) that the shutdown hook is called
by forcibly exiting the `Worker` and various points in the code with the
`WorkerWatcher` both disabled and enabled. Also, manually killed the driver
and verified that the `DriverRunner` killed the process and exited properly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BryanCutler/spark
DriverRunner-shutdown-hook-SPARK-13602
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11746.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11746
----
commit b0ce75ed40fe92f15aed1e30002ab51daaead950
Author: Bryan Cutler <[email protected]>
Date: 2016-03-15T23:40:17Z
[SPARK-13602] Added shutdown hook to DriverRunner and cleanup
commit 2402ecbdfe16b8cc7388cd93fd96140ef1fd8f69
Author: Bryan Cutler <[email protected]>
Date: 2016-03-15T23:41:41Z
Added tests for DriverRunner for finalization states and refit existing
test to work with changes
commit 96c3e7f516bf4c7b054bb5651fb777a3f0b0718a
Author: Bryan Cutler <[email protected]>
Date: 2016-03-15T23:54:37Z
Removed usage of AtomicRef for process, it is only shared with the shutdown
hook and if that is called, then does not matter at that point
commit e3f17f894636c9fdb8b451458ce9470af499703c
Author: Bryan Cutler <[email protected]>
Date: 2016-03-16T00:18:09Z
removed import of AtomicRef
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
