[
https://issues.apache.org/jira/browse/YARN-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231431#comment-15231431
]
Wangda Tan commented on YARN-4425:
----------------------------------
[~Naganarasimha], [~xinxianyin],
Read the doc and took a very high level look at the patch, sorry for the huge
delays. Some thoughts:
The new added policy seems like a backdoor of scheduler's capacity management:
when under "NON_EXCLUSIVE" mode, scheduler completely depends on configured
policy to decide who can get next resources.
And we need to make API of policy with more clear semantic: In existing
scheduler, resource will be allocated on requested partition, and only
request.partition = "" can get chance to allocate on other partitions when
non-exclusive criteria is met.
In the new API, resource could be allocated on any partition regardless of
requested partition. (depends on different policy implementation). Which will
be conflict to our existing APIs for node partitions.
To me, sharing resource between partitions itself is not a clear API:
You can say partition A has total resource = 100G, partition B has total
resource = 200G. But you cannot say: "under some conditions, partition A can
use idle resources from partition B" -- because partition is not the entity
which will consume resources.
Instead, it will be more clear to me to say:
1) Queue's shares of partitions could be dynamically adjusted, OR
2) Node's partition could be dynamically update
> Pluggable sharing policy for Partition Node Label resources
> -----------------------------------------------------------
>
> Key: YARN-4425
> URL: https://issues.apache.org/jira/browse/YARN-4425
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, resourcemanager
> Reporter: Naganarasimha G R
> Assignee: Naganarasimha G R
> Attachments: ResourceSharingPolicyForNodeLabelsPartitions-V1.pdf,
> ResourceSharingPolicyForNodeLabelsPartitions-V2.pdf,
> YARN-4425.20160105-1.patch
>
>
> As part of support for sharing NonExclusive Node Label partitions in
> YARN-3214, NonExclusive partitions are shared only to Default Partitions and
> also have fixed rule when apps in default partitions makes use of resources
> of any NonExclusive partitions.
> There are many scenarios where in we require pluggable policy like
> MutliTenant, Hierarchical etc.. where in each partition can determine when
> they want to share the resources to other paritions and when other partitions
> wants to use resources from others
> More details in the attached document.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
