[
https://issues.apache.org/jira/browse/SPARK-21655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116859#comment-16116859
]
Thomas Graves commented on SPARK-21655:
---------------------------------------
the yarn kill does work, but it does a kill signal on the container. Generally
the shutdown hooks allow things to unregister and cleanup ok, but we have seen
times this doesn't happen.
A spark kill command allows for things to shutdown and cleanup nicely. The
event queue can try to drain, we can make sure the history server url is proper
and do a stop like on a normal success case.
it also adds a nice api that users can use to kill. I have gotten the question
many times in the past from new users about how do I kill my application.
Having a spark cli to do this would be more convenient. It is already supported
across mesos and standalone so adding it here just makes it more consistent.
The underlying rpc connection can be used for other things in the future. Like
adding a better get status command, getting history info from the cli, pushing
new tokens, etc.
> Kill CLI for Yarn mode
> ----------------------
>
> Key: SPARK-21655
> URL: https://issues.apache.org/jira/browse/SPARK-21655
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.1.1
> Reporter: Jong Yoon Lee
> Priority: Minor
> Fix For: 2.1.1
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to how standalone and Mesos have the capability to safely shut down
> the spark application, there should be a way to safely shut down spark on
> Yarn mode. This will ensure a clean shutdown and unregistration from yarn.
> This is the design doc:
> https://docs.google.com/document/d/1QG8hITjLNi1D9dVR3b_hZkyrGm5FFm0u9M1KGM4y1Ak/edit?usp=sharing
> and I will upload the patch soon
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
