[
https://issues.apache.org/jira/browse/YARN-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427591#comment-16427591
]
Jason Lowe commented on YARN-8118:
----------------------------------
This is definitely not desirable in all cases. We often decommission a node in
order to gracefully remove it from service as soon as possible, and allowing
new containers to run on it will usually harm more than help that effort given
a sufficiently large cluster to run those containers elsewhere. If this is
added it should not change the default behavior as it exists today. It would
need to be either a config option for the whole cluster or a parameter as part
of the rmadmin command to gracefully decomm a specific node.
> Better utilize gracefully decommissioning node managers
> -------------------------------------------------------
>
> Key: YARN-8118
> URL: https://issues.apache.org/jira/browse/YARN-8118
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
> Affects Versions: 2.8.2
> Environment: * Google Compute Engine (Dataproc)
> * Java 8
> * Hadoop 2.8.2 using client-mode graceful decommissioning
> Reporter: Karthik Palaniappan
> Priority: Major
> Attachments: YARN-8118-branch-2.001.patch
>
>
> Proposal design doc with background + details (please comment directly on
> doc):
> [https://docs.google.com/document/d/1hF2Bod_m7rPgSXlunbWGn1cYi3-L61KvQhPlY9Jk9Hk/edit#heading=h.ab4ufqsj47b7]
> tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications
> to complete before shutting down, but they cannot run new containers from
> those in-progress applications. This is wasteful, particularly in
> environments where you are billed by resource usage (e.g. EC2).
> Proposal: YARN should schedule containers from in-progress applications on
> DECOMMISSIONING nodes, but should still avoid scheduling containers from new
> applications. That will make in-progress applications complete faster and let
> nodes decommission faster. Overall, this should be cheaper.
> I have a working patch without unit tests that's surprisingly just a few real
> lines of code (patch 001). If folks are happy with the proposal, I'll write
> unit tests and also write a patch targeted at trunk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
