Weiwei Yang commented on YARN-1042:

I am thinking about following approach, appreciate for suggestions : )

In ApplicationSubmissionContext class, add a new argument to indicate the 
container allocation rule in terms of affinity/anti-affinity. RM will follow 
the certain rules to allocate containers for this application. The argument is 
an instance of class ContainerAllocationRule(new), this class defines several 
types allocation rule, such as 

 * AFFINITY_REQUIRED: containers MUST be allocated on the same host/rack
 * AFFINITY_PREFERED: prefer to allocate containers on same host/rack if 
 * ANTI_AFFINITY_REQUIRED: containers MUST be allocated on different hosts/racks
 * ANTI_AFFINITY_PREFERED: prefer to allocate containers on different 
hosts/racks if possible

Each of these rules will have a handler on the RM side to add some control on 
container allocation. When a client submits an application with a certain 
ContainerAllocationRule to RM, this information will be added into 
ApplicationAttemptId (because the allocation rule is defined per application), 
when RM uses registered scheduler to allocate containers, it can retrieve the 
rule from ApplicationAttemptId and call particular handler during the 
allocation. The code can be added into 
SchedulerApplicationAttempt.pullNewlyAllocatedContainersAndNMTokens so to avoid 
modifying all schedulers.

> add ability to specify affinity/anti-affinity in container requests
> -------------------------------------------------------------------
>                 Key: YARN-1042
>                 URL: https://issues.apache.org/jira/browse/YARN-1042
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Arun C Murthy
>         Attachments: YARN-1042-demo.patch
> container requests to the AM should be able to request anti-affinity to 
> ensure that things like Region Servers don't come up on the same failure 
> zones. 
> Similarly, you may be able to want to specify affinity to same host or rack 
> without specifying which specific host/rack. Example: bringing up a small 
> giraph cluster in a large YARN cluster would benefit from having the 
> processes in the same rack purely for bandwidth reasons.

This message was sent by Atlassian JIRA

Reply via email to