[
https://issues.apache.org/jira/browse/YARN-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15938932#comment-15938932
]
Wangda Tan commented on YARN-6344:
----------------------------------
+1 for the proposal, and thanks [~kkaranasos] for sharing detailed analysis and
patch.
One question about the semantics of the new config:
{{yarn.scheduler.capacity.rack-locality-delay}}.
I think a valid config value should be >= {{node-locality-delay}}, correct? If
so, do you think is it better to make it to an additional value of
{{node-locality-delay}}, for example, node_locality_delay = x, and
rack_locality_delay = y makes actual_rack_locality_delay = x + y.
[~sunilg],
bq. But number of nodes may increase/decrease in a cluster at regular interval.
Why we are not considering a ratio to total-number-of-nodes? Did you met with
some other pitfalls in tat approach.
There's a discussion at YARN-4287 about this before,
https://issues.apache.org/jira/browse/YARN-4287?focusedCommentId=14980816&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14980816,
I think it is one thing we can do in the future, but for now an absolute
number of missed opportunity should be enough for most cases.
> Rethinking OFF_SWITCH locality in CapacityScheduler
> ---------------------------------------------------
>
> Key: YARN-6344
> URL: https://issues.apache.org/jira/browse/YARN-6344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
> Reporter: Konstantinos Karanasos
> Assignee: Konstantinos Karanasos
> Attachments: YARN-6344.001.patch
>
>
> When relaxing locality from node to rack, the {{node-locality-parameter}} is
> used: when scheduling opportunities for a scheduler key are more than the
> value of this parameter, we relax locality and try to assign the container to
> a node in the corresponding rack.
> On the other hand, when relaxing locality to off-switch (i.e., assign the
> container anywhere in the cluster), we are using a {{localityWaitFactor}},
> which is computed based on the number of outstanding requests for a specific
> scheduler key, which is divided by the size of the cluster.
> In case of applications that request containers in big batches (e.g.,
> traditional MR jobs), and for relatively small clusters, the
> localityWaitFactor does not affect relaxing locality much.
> However, in case of applications that request containers in small batches,
> this load factor takes a very small value, which leads to assigning
> off-switch containers too soon. This situation is even more pronounced in big
> clusters.
> For example, if an application requests only one container per request, the
> locality will be relaxed after a single missed scheduling opportunity.
> The purpose of this JIRA is to rethink the way we are relaxing locality for
> off-switch assignments.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
