[
https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990765#comment-13990765
]
Carlo Curino commented on YARN-2022:
------------------------------------
Sunil, I completely support this idea, I was actually looking into it just
recently (since during some stress-test, corner-case experiments I saw large
numbers of jobs getting killed because of the reason you point out).
There a couple of tricky things however:
* We don't have information on which container is the AM one readily available
in RMContainer/Container, often it is the first container, but not 100%
guaranteed (next attempts might have a different container_id). Maybe we could
add this as part of the tracked information.
* We should think about corner cases, e.g., if the cluster capacity is reduced
and we are not killing AMs, the entire queue worth of resources is full of AMs
with no room to make any progress... This and other similar scenarios should be
thought through a bit more.
Maybe you could add it and have a tunable to turn-it on or off and we can gain
some experience with it. If you provide a patch, I would be happy to review it
and run tests in our test clusters and report on what I see.
> Preempting an Application Master container can be kept as least priority when
> multiple applications are marked for preemption by
> ProportionalCapacityPreemptionPolicy
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-2022
> URL: https://issues.apache.org/jira/browse/YARN-2022
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
> Affects Versions: 2.4.0
> Reporter: Sunil G
> Assignee: Sunil G
>
> Cluster Size = 16GB [2NM's]
> Queue A Capacity = 50%
> Queue B Capacity = 50%
> Consider there are 3 applications running in Queue A which has taken the full
> cluster capacity.
> J1 = 2GB AM + 1GB * 4 Maps
> J2 = 2GB AM + 1GB * 4 Maps
> J3 = 2GB AM + 1GB * 2 Maps
> Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ].
> Currently in this scenario, Jobs J3 will get killed including its AM.
> It is better if AM can be given least priority among multiple applications.
> In this same scenario, map tasks from J3 and J4 can be preempted.
> Later when cluster is free, maps can be allocated to these Jobs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
