[
https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043104#comment-14043104
]
Sunil G commented on YARN-2022:
-------------------------------
Thank You [~mayank_bansal] for the review. I understood the scenario which you
explained and split-up of 50% should be 5% + 45%.
bq.Resource maxAMCapacity =
Resources.multiply(Resources.multiply(clusterResource,
qT.leafQueue.getAbsoluteMaximumCapacity()),
qT.leafQueue.getMaxAMResourcePerQueuePercent());
Yes, I used *getMaxAMResourcePerQueuePercent* with *getAbsoluteMaximumCapacity*
to get the maxAMCapacity in a Queue. *getAbsoluteCapacity* was a choice instead
of *getAbsoluteMaximumCapacity*.
But when I checked LeafQueue for *maxActiveApplications* calculation, I found
that getAbsoluteMaximumCapacity was used there.
We can use *getAbsoluteCapacity* here for calculating the AMCapacity for a
Queue. With this approach, there are chances that more AM will be killed under
scenarios like below.
For eg, Queue A with Capacity 25%, Max capacity 100%, Queue B with capacity
75%, AM Percent is 0.5.
If QueueA is using 100% of cluster with 50% of AM (it is possible with
different users) then with a big demand from QueueB, 37.5% of AMs from QueueA
also can be preempted. I feel this is fine as you suggested.
Could I go along by changing maxAMCapacity w.r.t getAbsoluteCapacity?
> Preempting an Application Master container can be kept as least priority when
> multiple applications are marked for preemption by
> ProportionalCapacityPreemptionPolicy
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-2022
> URL: https://issues.apache.org/jira/browse/YARN-2022
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 2.4.0
> Reporter: Sunil G
> Assignee: Sunil G
> Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch,
> YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, YARN-2022.6.patch,
> YARN-2022.7.patch, Yarn-2022.1.patch
>
>
> Cluster Size = 16GB [2NM's]
> Queue A Capacity = 50%
> Queue B Capacity = 50%
> Consider there are 3 applications running in Queue A which has taken the full
> cluster capacity.
> J1 = 2GB AM + 1GB * 4 Maps
> J2 = 2GB AM + 1GB * 4 Maps
> J3 = 2GB AM + 1GB * 2 Maps
> Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ].
> Currently in this scenario, Jobs J3 will get killed including its AM.
> It is better if AM can be given least priority among multiple applications.
> In this same scenario, map tasks from J3 and J2 can be preempted.
> Later when cluster is free, maps can be allocated to these Jobs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
