Jonathan Hung created YARN-9730:
-----------------------------------

             Summary: Support forcing configured partitions to be exclusive 
based on app node label
                 Key: YARN-9730
                 URL: https://issues.apache.org/jira/browse/YARN-9730
             Project: Hadoop YARN
          Issue Type: Task
            Reporter: Jonathan Hung
            Assignee: Jonathan Hung


Use case: queue X has all of its workload in non-default (exclusive) partition 
P (by setting app submission context's node label set to P). Node in partition 
Q != P heartbeats to RM. Capacity scheduler loops through every application in 
X, and every scheduler key in this application, and fails to allocate each time 
since the app's requested label and the node's label don't match. This causes 
huge performance degradation when number of apps in X is large.

To fix the issue, allow RM to configure partitions as "forced-exclusive". If 
partition P is "forced-exclusive", then:
 * If app sets its submission context's node label to P, all its resource 
requests will be overridden to P
 * If app sets its submission context's node label Q, any of its resource 
requests whose labels are P will be overridden to Q
 * In the scheduler, we add apps with node label expression P to a separate 
data structure. When a node in partition P heartbeats to scheduler, we only try 
to schedule apps in this data structure. When a node in partition Q heartbeats 
to scheduler, we schedule the rest of the apps as normal.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Reply via email to