Karthik Palaniappan created YARN-8118:
-----------------------------------------
Summary: Better utilize gracefully decommissioning node managers
Key: YARN-8118
URL: https://issues.apache.org/jira/browse/YARN-8118
Project: Hadoop YARN
Issue Type: Sub-task
Components: yarn
Affects Versions: 2.8.2
Environment: * Google Compute Engine (Dataproc)
* Java 8
* Hadoop 2.8.2 using client-mode graceful decommissioning
Reporter: Karthik Palaniappan
Proposal design doc with background + details: (will link – please comment on
that doc)
tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications
to complete before shutting down, but they cannot run new containers from those
in-progress applications. This is wasteful, particularly in environments where
you are billed by resource usage (e.g. EC2).
Proposal: YARN should schedule containers from in-progress applications on
DECOMMISSIONING nodes, but should still avoid scheduling containers from new
applications. That will make in-progress applications complete faster and let
nodes decommission faster. Overall, this should be cheaper.
I have a working patch without unit tests (will attach) that's surprisingly
just a few real lines of code. If folks are happy with the proposal, I'll write
unit tests and also write a patch targeted at trunk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
