Ben Podgursky commented on YARN-2154:

Hi, I've been watching this ticket since this has been a big problem on our 
cluster.    Karthik, your last comment concerned me because it seems like it 
will violate one of the things FairScheduler is supposed to promise:

"Unlike the default Hadoop scheduler, which forms a queue of jobs, this lets 
short jobs finish in reasonable time while not starving long jobs.'


"When other jobs are submitted, tasks slots that free up are assigned to the 
new jobs, so that each job gets roughly the same amount of CPU time. "

If a job can't even get an AM, it surely won't be able to get an equitable 
portion of slots for Maps and Reduces -- being able to get small jobs through 
quickly even when the pool has huge jobs running is the reason we use 

Sorry if I'm misunderstanding that comment.

> FairScheduler: Improve preemption to preempt only those containers that would 
> satisfy the incoming request
> ----------------------------------------------------------------------------------------------------------
>                 Key: YARN-2154
>                 URL: https://issues.apache.org/jira/browse/YARN-2154
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.4.0
>            Reporter: Karthik Kambatla
>            Assignee: Arun Suresh
>            Priority: Critical
>         Attachments: YARN-2154.1.patch
> Today, FairScheduler uses a spray-gun approach to preemption. Instead, it 
> should only preempt resources that would satisfy the incoming request. 

This message was sent by Atlassian JIRA

Reply via email to