[
https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos resolved YARN-2877.
------------------------------------------
Resolution: Resolved
Hadoop Flags: Reviewed
Release Note:
With this JIRA we are introducing distributed scheduling in YARN.
In particular, we make the following contributions:
- Introduce the notion of container types. GUARANTEED containers follow the
semantics of the existing YARN containers. OPPORTUNISTIC ones can be seen as
lower priority containers, and can be preempted in order to make space for
GUARANTEED containers to run.
- Queuing of tasks at the NMs. This enables us to send more containers in an NM
than its available resources. At the moment we are allowing queuing of
OPPORTUNISTIC containers. Once resources become available at the NM, such
containers can immediately start their execution.
- Introduce the AMRMProxy. This is a service running at each node, intercepting
the requests between the AM and the RM. It is instrumental for both distributed
scheduling and YARN Federation (YARN-2915).
- Enable distributed scheduling. To minimize their allocation latency,
OPPORTUNISTIC containers are dispatched immediately to NMs in a distributed
fashion by using the AMRMProxy of the node where the corresponding AM resides,
without needing to go through the ResourceManager.
All the functionality introduced in this JIRA is disabled by default, so it
will not affect the behavior of existing applications.
We have introduced parameters in YarnConfiguration to enable NM queuing
(yarn.nodemanager.container-queuing-enabled), distributed scheduling
(yarn.distributed-scheduling.enabled) and the AMRMProxy service
(yarn.nodemanager.amrmproxy.enable).
AMs currently need to specify the type of container to be requested for each
task. We are in the process of adding in the MapReduce AM the ability to
randomly request OPPORTUNISTIC containers for a specified percentage of a job's
tasks, so that users can experiment with the new features.
Target Version/s: 3.0.0-alpha1
> Extend YARN to support distributed scheduling
> ---------------------------------------------
>
> Key: YARN-2877
> URL: https://issues.apache.org/jira/browse/YARN-2877
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, resourcemanager
> Reporter: Sriram Rao
> Assignee: Konstantinos Karanasos
> Attachments: distributed-scheduling-design-doc_v1.pdf
>
>
> This is an umbrella JIRA that proposes to extend YARN to support distributed
> scheduling. Briefly, some of the motivations for distributed scheduling are
> the following:
> 1. Improve cluster utilization by opportunistically executing tasks otherwise
> idle resources on individual machines.
> 2. Reduce allocation latency. Tasks where the scheduling time dominates
> (i.e., task execution time is much less compared to the time required for
> obtaining a container from the RM).
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
