Sunil G commented on YARN-2004:

Hi Jason, thank you for sharing the thoughts.

In one way, we need not have to think abt headroom and userlimit. Still I would 
like to share 2 scenarios

1. Similar to MAPREDUCE-314. A job j1 is submitted with lower priority and 
finished its map tasks, reducers are running. later j2 and j3 came in and took 
over cluster resources. if a map is failed, by loosing some map o/p, there are 
no chances of getting a resource for j1 till j2 and j3 releases resources and 
not allocating it. In a -ve scenario, j1 will starve for much longer. This was 
one of the intention to temporarily pause demand from j2 and j4 for a while and 
spare some resources for j1.

2. User Limit: Assume the factor is 25, and 4 users can take 25% each from 
cluster. 5th user has to wait. Assume the highest priority app is submitted by 
5th user. He may not get resources untill demand from first 4 users(for 
existing apps) are over. Do you feel this is to be handled?

> Priority scheduling support in Capacity scheduler
> -------------------------------------------------
>                 Key: YARN-2004
>                 URL: https://issues.apache.org/jira/browse/YARN-2004
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: 0001-YARN-2004.patch
> Based on the priority of the application, Capacity Scheduler should be able 
> to give preference to application while doing scheduling.
> Comparator<FiCaSchedulerApp> applicationComparator can be changed as below.   
> 1.    Check for Application priority. If priority is available, then return 
> the highest priority job.
> 2.    Otherwise continue with existing logic such as App ID comparison and 
> then TimeStamp comparison.

This message was sent by Atlassian JIRA

Reply via email to