GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/1643
[SPARK-1740] [PySpark] kill the python worker
Kill only the python worker related to cancelled tasks.
The daemon will start a background thread to monitor all the opened sockets
for all workers. If the socket is closed by JVM, this thread will kill the
worker.
When an task is cancelled, the socket to worker will be closed, then the
worker will be killed by deamon.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark kill
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1643.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1643
----
commit 9c62c675e1761532c171aaba4f030aa6b28f311d
Author: Davies Liu <[email protected]>
Date: 2014-07-30T00:07:25Z
kill the python worker
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
