GitHub user piaozhexiu opened a pull request:
https://github.com/apache/spark/pull/5343
[SPARK-6692][YARN] Make it possible to kill AM in YARN cluster mode
I understand that the yarn-cluster mode is designed for fire-and-forget
model; therefore, terminating the yarn client doesn't kill AM.
However, it is very common that users submit Spark jobs via job scheduler
(e.g. Apache Oozie) or remote job server (e.g. Netflix Genie) where it is
expected that killing the yarn client will terminate AM.
It is true that the yarn-client mode can be used in such cases. But then,
the yarn client sometimes needs lots of heap memory for big jobs if it runs in
the yarn-client mode. In fact, the yarn-cluster mode is ideal for big jobs
because AM can be given arbitrary heap memory unlike the yarn client. So it
would be very useful to make it possible to kill AM even in the yarn-cluster
mode.
In addition, Spark jobs often become zombie jobs if users ctrl-c them as
soon as they're accepted (but not yet running). Although they're eventually
shutdown after AM timeout, it would be nice if AM could immediately get killed
in such cases too.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/piaozhexiu/spark SPARK-6692
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5343.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5343
----
commit ef2793c497d7b8f3baf03a47cbbdf6845e31f05e
Author: Cheolsoo Park <[email protected]>
Date: 2015-04-03T06:35:24Z
Make it possible to kill AM in YARN cluster mode when the client is
terminated
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
