[
https://issues.apache.org/jira/browse/YARN-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-2915:
---------------------------------
Release Note: A federation-based approach to transparently scale a single
YARN cluster to tens of thousands of nodes, by federating multiple YARN
standalone clusters (sub-clusters). The applications running in this federated
environment will see a single massive YARN cluster and will be able to schedule
tasks on any node of the federated cluster. Under the hood, the federation
system will negotiate with sub-clusters ResourceManagers and provide resources
to the application. The goal is to allow an individual job to “span”
sub-clusters seamlessly. (was: A federation-based approach to transparently
scale a single YARN cluster to tens of thousands of nodes, by federating
multiple YARN standalone clusters.The proposed approach is to divide a large
(10-100k nodes) cluster into smaller units called sub-clusters, each with its
own YARN RM and compute nodes. The federation system will stitch these
sub-clusters together and make them appear as one large YARN cluster to the
applications. The applications running in this federated environment will see a
single massive YARN cluster and will be able to schedule tasks on any node of
the federated cluster. Under the hood, the federation system will negotiate
with sub-clusters resource managers and provide resources to the application.
The goal is to allow an individual job to “span” sub-clusters seamlessly.)
> Enable YARN RM scale out via federation using multiple RM's
> -----------------------------------------------------------
>
> Key: YARN-2915
> URL: https://issues.apache.org/jira/browse/YARN-2915
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, resourcemanager
> Reporter: Sriram Rao
> Assignee: Subru Krishnan
> Labels: federation
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf,
> Federation-BoF.pdf, YARN-Federation-Hadoop-Summit_final.pptx,
> Yarn_federation_design_v1.pdf, federation-prototype.patch
>
>
> This is an umbrella JIRA that proposes to scale out YARN to support large
> clusters comprising of tens of thousands of nodes. That is, rather than
> limiting a YARN managed cluster to about 4k in size, the proposal is to
> enable the YARN managed cluster to be elastically scalable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
