Mayank Bansal commented on YARN-2022:

hi [~sunilg] 

Thanks for the patch.

There is small addition we need to do in the approach which is as follows

we need to consider below parameters into account
yarn.scheduler.capacity.maximum-am-resource-percent / 

It would be if user have set  
yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent in the queue 
then we can not preempt AM even if we didn't reach to full resource need from 
the queue.

If user didn't set that queue level setting then we need to check if we are not 
avoiding yarn.scheduler.capacity.maximum-am-resource-percent constraint as well 

if these two constraints are not avoided and still we have some ams which we 
need to kill then yes we can go with the approach you put in your patch.


> Preempting an Application Master container can be kept as least priority when 
> multiple applications are marked for preemption by 
> ProportionalCapacityPreemptionPolicy
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>                 Key: YARN-2022
>                 URL: https://issues.apache.org/jira/browse/YARN-2022
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, 
> YARN-2022.3.patch, Yarn-2022.1.patch
> Cluster Size = 16GB [2NM's]
> Queue A Capacity = 50%
> Queue B Capacity = 50%
> Consider there are 3 applications running in Queue A which has taken the full 
> cluster capacity. 
> J1 = 2GB AM + 1GB * 4 Maps
> J2 = 2GB AM + 1GB * 4 Maps
> J3 = 2GB AM + 1GB * 2 Maps
> Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ].
> Currently in this scenario, Jobs J3 will get killed including its AM.
> It is better if AM can be given least priority among multiple applications. 
> In this same scenario, map tasks from J3 and J2 can be preempted.
> Later when cluster is free, maps can be allocated to these Jobs.

This message was sent by Atlassian JIRA

Reply via email to