Karthik Palaniappan created YARN-8118: -----------------------------------------
Summary: Better utilize gracefully decommissioning node managers Key: YARN-8118 URL: https://issues.apache.org/jira/browse/YARN-8118 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Affects Versions: 2.8.2 Environment: * Google Compute Engine (Dataproc) * Java 8 * Hadoop 2.8.2 using client-mode graceful decommissioning Reporter: Karthik Palaniappan Proposal design doc with background + details: (will link – please comment on that doc) tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications to complete before shutting down, but they cannot run new containers from those in-progress applications. This is wasteful, particularly in environments where you are billed by resource usage (e.g. EC2). Proposal: YARN should schedule containers from in-progress applications on DECOMMISSIONING nodes, but should still avoid scheduling containers from new applications. That will make in-progress applications complete faster and let nodes decommission faster. Overall, this should be cheaper. I have a working patch without unit tests (will attach) that's surprisingly just a few real lines of code. If folks are happy with the proposal, I'll write unit tests and also write a patch targeted at trunk. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org