[
https://issues.apache.org/jira/browse/YARN-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197956#comment-15197956
]
Vinod Kumar Vavilapalli commented on YARN-2915:
-----------------------------------------------
One thing that occurred to me in an offline conversation with [~subru] and
[~leftnoteasy] is about the modeling of queues and their shares in different
sub-clusters.
As seems to be already proposed, it is very desirable to have a unified *logic
queues* that are applicable across all sub-clusters.
With unified logical queues, looks like there are some proposals for ways of
how resources can get sub-divided amongst different sub-clusters. But to me,
they already map to an existing concept in YARN - *Node Partitions* /
node-labels !
Essentially you have *one YARN cluster* -> *multiple sub-clusters* -> *each
sub-cluster with multiple node-partitions*. This can further be extended to
more levels. For e.g. we can unify rack also under the same concept.
The advantage of unifying this with node-partitions is that we can have
- one single administrative view philosophy of sub-clusters, node-partitions,
racks etc
- unified configuration mechanisms: Today we support centralized and
distributed node-partition mechanisms, exclusive / non-exclusive access etc.
- unified queue-sharing models - today we already can assign X% of a
node-partition to a queue. This way we can, again, reuse existing concepts,
mental models and allocation policies - instead of creating specific policies
for sub-cluster sharing like the user-based share that is proposed.
We will have to dig deeper into the details, but it seems to me that
node-partition and sub-cluster are equivalence classes except for the fact that
two sub-clusters report to two different RMs (physically / implementation wise)
which isn't the case today with node-partitions.
Thoughts? /cc [~curino] [~chris.douglas]
> Enable YARN RM scale out via federation using multiple RM's
> -----------------------------------------------------------
>
> Key: YARN-2915
> URL: https://issues.apache.org/jira/browse/YARN-2915
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, resourcemanager
> Reporter: Sriram Rao
> Assignee: Subru Krishnan
> Attachments: FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf,
> Federation-BoF.pdf, Yarn_federation_design_v1.pdf, federation-prototype.patch
>
>
> This is an umbrella JIRA that proposes to scale out YARN to support large
> clusters comprising of tens of thousands of nodes. That is, rather than
> limiting a YARN managed cluster to about 4k in size, the proposal is to
> enable the YARN managed cluster to be elastically scalable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
