[
https://issues.apache.org/jira/browse/YARN-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16919033#comment-16919033
]
Jonathan Hung commented on YARN-9730:
-------------------------------------
Thanks for looking [~haibochen]. To your first point, yes. A partition X is
"exclusive-enforced" means an app has access to X if and only if its
appSubmissionContext node label is X. (Perhaps we should rename this
exclusive-enforced terminology to avoid confusion with the existing "exclusive"
node label terminology)
To your second point, it seems appAMNodePartitionName refers only to the label
expression for the app's AM resource request which is different from the
appSubmissionContext label expression. So we cannot reuse it.
Node label expression for an app still makes sense for those not submitted to
an enforced partition. The behavior will just fallback to existing behavior
(IIRC, if a resource request for an app has no label expression, it will use
the app's node label expression.)
> Support forcing configured partitions to be exclusive based on app node label
> -----------------------------------------------------------------------------
>
> Key: YARN-9730
> URL: https://issues.apache.org/jira/browse/YARN-9730
> Project: Hadoop YARN
> Issue Type: Task
> Reporter: Jonathan Hung
> Assignee: Jonathan Hung
> Priority: Major
> Labels: release-blocker
> Attachments: YARN-9730.001.patch, YARN-9730.002.patch
>
>
> Use case: queue X has all of its workload in non-default (exclusive)
> partition P (by setting app submission context's node label set to P). Node
> in partition Q != P heartbeats to RM. Capacity scheduler loops through every
> application in X, and every scheduler key in this application, and fails to
> allocate each time since the app's requested label and the node's label don't
> match. This causes huge performance degradation when number of apps in X is
> large.
> To fix the issue, allow RM to configure partitions as "forced-exclusive". If
> partition P is "forced-exclusive", then:
> * 1a. If app sets its submission context's node label to P, all its resource
> requests will be overridden to P
> * 1b. If app sets its submission context's node label Q, any of its resource
> requests whose labels are P will be overridden to Q
> * 2. In the scheduler, we add apps with node label expression P to a
> separate data structure. When a node in partition P heartbeats to scheduler,
> we only try to schedule apps in this data structure. When a node in partition
> Q heartbeats to scheduler, we schedule the rest of the apps as normal.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
