[ 
https://issues.apache.org/jira/browse/YARN-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277174#comment-14277174
 ] 

Wei Yan commented on YARN-3054:
-------------------------------

Hi, [~peng.zhang]. Firstly, FairScheduler will check whether the usage is over 
fairness.
{code}
  private boolean preemptContainerPreCheck() {
    return parent.getPolicy().checkIfUsageOverFairShare(getResourceUsage(),
        getFairShare());
  }
{code}

bq. Mapreduce jobs can get additional resources when others are idle.
I'm not sure what your "idle" meaning here. But in YARN, one queue can take the 
over-fairshare resource, if the resources are not used by other queues. And in 
FairScheduler, each queue has "steady" fairshare and "dynamic" fairshare. For 
example, if we have two queues (Q1 and Q2), both with weight 1. So Q1's steady 
share is 50%, and Q2 is also 50%. Assume only Q1 has jobs and no job submitted 
to Q2, Q1's dynamic fairness is 100% and Q2 is 0. The dynamic fairshare 
calculation only considers "active" queues.

bq. Mapreduce jobs for one user in one queue can still progress with its min 
share when others preempt resources back.
As I said above, each queue is guaranted with minshare and fairshare. That 
means, some jobs can still move on. We cannot assign a minshare to each job. 
Otherwise, the job with multiple concurrent jobs may take over the cluster.

> Preempt policy in FairScheduler may cause mapreduce job never finish
> --------------------------------------------------------------------
>
>                 Key: YARN-3054
>                 URL: https://issues.apache.org/jira/browse/YARN-3054
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.6.0
>            Reporter: Peng Zhang
>
> Preemption policy is related with schedule policy now. Using comparator of 
> schedule policy to find preemption candidate cannot guarantee a subset of 
> containers never be preempted. And this may cause tasks to be preempted 
> periodically before they finish. So job cannot make any progress. 
> I think preemption in YARN should got below assurance:
> 1. Mapreduce jobs can get additional resources when others are idle;
> 2. Mapreduce jobs for one user in one queue can still progress with its min 
> share when others preempt resources back.
> Maybe always preempt the latest app and container can get this? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to