[
https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985661#comment-15985661
]
Wangda Tan commented on YARN-5881:
----------------------------------
Thanks [~sunilg] for the POC patch, general approach looks good, several
structural/cosmetic suggestions:
1) For the quotas (Like effective min/max, configured min/max, etc.) I think it
is better put all of then to one class. Because it can handle following things
better: a. for not existed label, it can return a zero resource instead of
null. b. can have a separate lock.
What I'm thinking now is moving common methods / fields from ResourceUsage to a
common class, and leave all type-specifc methods (such as setCachedUsed) in the
subclass.
And we can create a a new class (Such as QueueResourceQuotas), which extend the
base class, but with some methods to read/write queue resource quota types.
Downside of the approach is, resources associated to one partition is
determined by ResourceUsage#ResourceType. Without lots of changes, for each
partition, it will include many unnecessary resource types (for example
used_resource is unnecessary for ResourceQuotas). It looks not a big problem to
me since accessing items in an array is very fast and number of partitions is
limited.
2) Can we move {{loadResourceConstraintByLabelsFromConf}} from {{CSQueueUtils}}
to AbstractCSQueue? Since it is more related to the queue and its parent queue,
with this change lots of input parameters of
{{loadResourceConstraintByLabelsFromConf}} can be removed.
3) For set effective min/max Resource, do you think we can only do it inside
{{LeafQueue/ParentQueue#updateClusterResource}}? Now
{{computeEffectiveResourcesForAllQueues}} creates a separate recursive call, I
think it will be better to put all queue-resource-related changes inside
{{LeafQueue/ParentQueue#updateClusterResource}}.
Please let me know your thoughts. And for the next patch, it's better if you
can update all scheduler paths to use getEffective*.
> Enable configuration of queue capacity in terms of absolute resources
> ---------------------------------------------------------------------
>
> Key: YARN-5881
> URL: https://issues.apache.org/jira/browse/YARN-5881
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Sean Po
> Assignee: Wangda Tan
> Attachments:
> YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf,
> YARN-5881.v0.patch, YARN-5881.v1.patch
>
>
> Currently, Yarn RM supports the configuration of queue capacity in terms of a
> proportion to cluster capacity. In the context of Yarn being used as a public
> cloud service, it makes more sense if queues can be configured absolutely.
> This will allow administrators to set usage limits more concretely and
> simplify customer expectations for cluster allocation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
