[ 
https://issues.apache.org/jira/browse/YARN-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Palaniappan updated YARN-8118:
--------------------------------------
    Description: 
Proposal design doc with background + details (please comment directly on doc): 
[https://docs.google.com/document/d/1hF2Bod_m7rPgSXlunbWGn1cYi3-L61KvQhPlY9Jk9Hk/edit#heading=h.ab4ufqsj47b7]

tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications 
to complete before shutting down, but they cannot run new containers from those 
in-progress applications. This is wasteful, particularly in environments where 
you are billed by resource usage (e.g. EC2).

Proposal: YARN should schedule containers from in-progress applications on 
DECOMMISSIONING nodes, but should still avoid scheduling containers from new 
applications. That will make in-progress applications complete faster and let 
nodes decommission faster. Overall, this should be cheaper.

I have a working patch without unit tests that's surprisingly just a few real 
lines of code (patch 001). If folks are happy with the proposal, I'll write 
unit tests and also write a patch targeted at trunk.

  was:
Proposal design doc with background + details (please comment directly on doc): 
[https://docs.google.com/document/d/1hF2Bod_m7rPgSXlunbWGn1cYi3-L61KvQhPlY9Jk9Hk/edit#heading=h.ab4ufqsj47b7]

tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications 
to complete before shutting down, but they cannot run new containers from those 
in-progress applications. This is wasteful, particularly in environments where 
you are billed by resource usage (e.g. EC2).

Proposal: YARN should schedule containers from in-progress applications on 
DECOMMISSIONING nodes, but should still avoid scheduling containers from new 
applications. That will make in-progress applications complete faster and let 
nodes decommission faster. Overall, this should be cheaper.

I have a working patch without unit tests (will attach) that's surprisingly 
just a few real lines of code. If folks are happy with the proposal, I'll write 
unit tests and also write a patch targeted at trunk.


> Better utilize gracefully decommissioning node managers
> -------------------------------------------------------
>
>                 Key: YARN-8118
>                 URL: https://issues.apache.org/jira/browse/YARN-8118
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn
>    Affects Versions: 2.8.2
>         Environment: * Google Compute Engine (Dataproc)
>  * Java 8
>  * Hadoop 2.8.2 using client-mode graceful decommissioning
>            Reporter: Karthik Palaniappan
>            Priority: Major
>         Attachments: YARN-8118-branch-2.001.patch
>
>
> Proposal design doc with background + details (please comment directly on 
> doc): 
> [https://docs.google.com/document/d/1hF2Bod_m7rPgSXlunbWGn1cYi3-L61KvQhPlY9Jk9Hk/edit#heading=h.ab4ufqsj47b7]
> tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications 
> to complete before shutting down, but they cannot run new containers from 
> those in-progress applications. This is wasteful, particularly in 
> environments where you are billed by resource usage (e.g. EC2).
> Proposal: YARN should schedule containers from in-progress applications on 
> DECOMMISSIONING nodes, but should still avoid scheduling containers from new 
> applications. That will make in-progress applications complete faster and let 
> nodes decommission faster. Overall, this should be cheaper.
> I have a working patch without unit tests that's surprisingly just a few real 
> lines of code (patch 001). If folks are happy with the proposal, I'll write 
> unit tests and also write a patch targeted at trunk.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to