[
https://issues.apache.org/jira/browse/YARN-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241962#comment-15241962
]
Wangda Tan commented on YARN-4425:
----------------------------------
[~Naganarasimha],
Thanks, let me rephrase what we shouldn't change and what we could change:
*Following protocol that shouldn't change:*
- Allocated resource to app should on the same partition of what requested by
the app.
We definitely should avoid defining protocol to allow sharing between
partitions, that's not intuitive.
If you look at non-exclusive node partition again, it is not resource sharing
between partition. We previously decided that empty node-label-expression means
"prefers on default node partition, but anywhere else is also acceptable". I
admit this is not easy to understand as well, but at least better than "sharing
between partition". If I have chance to re-do the non-exclusive partition, I
will not define protocols like that.
I suggest you can take look at YARN-4902, the better API to support
non-exclusive node partition should be:
{code}
Delayed-OR [
- Give me default partition,
(Wait for x mins)
- Any partition is acceptable.
]
{code}
Similarly, if you want to support "only partition-B can share partition-A's
idle resource" should be:
{code}
Delayed-OR [
- Give me partition=B,
(Wait for x mins)
- partition=A is also acceptable
]
{code}
*What we could change:*
- Adjust queue's shares on partitions based on policy
I don't think this will be harder than new sharing policy proposed by this
JIRA. If we want to do this internally in YARN, a couple of
changes/refactorings of CS required to support dynamically configuration of
queue capacities.
- Update nodes' partition externally based on policy:
bq. So practically this needs to be handled externally instead based on policy
i try to acheive the same thing transparently from inside. Thoughts?
You can use centralized-delegation to update partitions inside YARN.
> Pluggable sharing policy for Partition Node Label resources
> -----------------------------------------------------------
>
> Key: YARN-4425
> URL: https://issues.apache.org/jira/browse/YARN-4425
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, resourcemanager
> Reporter: Naganarasimha G R
> Assignee: Naganarasimha G R
> Attachments: ResourceSharingPolicyForNodeLabelsPartitions-V1.pdf,
> ResourceSharingPolicyForNodeLabelsPartitions-V2.pdf,
> YARN-4425.20160105-1.patch
>
>
> As part of support for sharing NonExclusive Node Label partitions in
> YARN-3214, NonExclusive partitions are shared only to Default Partitions and
> also have fixed rule when apps in default partitions makes use of resources
> of any NonExclusive partitions.
> There are many scenarios where in we require pluggable policy like
> MutliTenant, Hierarchical etc.. where in each partition can determine when
> they want to share the resources to other paritions and when other partitions
> wants to use resources from others
> More details in the attached document.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
