Sunil G commented on YARN-2022:

Thank you very much Carlo for the review. 
As per your concern about AM Container priority, I am using a static final 
variable named AM_CONTAINER_PRIORITY from RMAppAttemptImpl to check whether a 
container is AM or not.
As per my code review, this variable is not been set by user [RM only uses this 
to create an AM container Resource Request]. Hence there is no much problem in 
using the same.

Secondly for the corner cases, I agree with your point. In a specific corner 
case it is possible that 100% AM can take over a queue.
1. maximum-am-resource-percent is in cluster level and we can get maximum 
runnable applications. Actual total running applications count can also be 
fetched from all leaf queues.
   With these two, a checkpoint can be derived as you have mentioned.
2. user-limit-factor will set a user limit quota among total resources for each 
user. If preemption has to be done among applications, currently only 
application timestamp is considered [reverse order].
   So how this factor can help in giving a checkpoint for saving AM. Could you 
please share your thoughts on this point.

I will work on defining checkpoint for saving AM and will update. Meanwhile 
please check whether my explanation is in-line with you thoughts. Thank you.

> Preempting an Application Master container can be kept as least priority when 
> multiple applications are marked for preemption by 
> ProportionalCapacityPreemptionPolicy
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>                 Key: YARN-2022
>                 URL: https://issues.apache.org/jira/browse/YARN-2022
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: Yarn-2022.1.patch
> Cluster Size = 16GB [2NM's]
> Queue A Capacity = 50%
> Queue B Capacity = 50%
> Consider there are 3 applications running in Queue A which has taken the full 
> cluster capacity. 
> J1 = 2GB AM + 1GB * 4 Maps
> J2 = 2GB AM + 1GB * 4 Maps
> J3 = 2GB AM + 1GB * 2 Maps
> Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ].
> Currently in this scenario, Jobs J3 will get killed including its AM.
> It is better if AM can be given least priority among multiple applications. 
> In this same scenario, map tasks from J3 and J2 can be preempted.
> Later when cluster is free, maps can be allocated to these Jobs.

This message was sent by Atlassian JIRA

Reply via email to