Yufei Gu commented on YARN-7859:

I kind of understand the issue. Here is an example, please let me know if it is 
the case you talked about. Let's say there are two and only two queues under 
the root, q1 and q2. q1 has a long-running large job which occupy 60% fair 
share usage, and people keep submit short-time and small jobs to q2, which 
never get up to 60% fair share usage. In that case, q1 won't be scheduled for a 
while until the job inside the q1 finished or q2 usage increase faster enough 
to go over 60%, even there are available resources. That's a valid issue, not a 
common case though. There is some assumptions, like the submission of jobs in 
q1 is faster than scheduling interval, and usage changes in q1 is slower than 

I can think about several ways to mitigate the issue in current implementation. 
None of them solve that issue though. The promising one is the global 
scheduling, which is still under development. 

> New feature: add queue scheduling deadLine in fairScheduler.
> ------------------------------------------------------------
>                 Key: YARN-7859
>                 URL: https://issues.apache.org/jira/browse/YARN-7859
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: fairscheduler
>    Affects Versions: 3.0.0
>            Reporter: wangwj
>            Assignee: wangwj
>            Priority: Major
>              Labels: fairscheduler, features, patch
>             Fix For: 3.0.0
>         Attachments: YARN-7859-v1.patch, log, screenshot-1.png, 
> screenshot-3.png
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>      As everyone knows.In FairScheduler the phenomenon of queue scheduling 
> starvation often occurs when the number of cluster jobs is large.The App in 
> one or more queue are pending.So I have thought a way to solve this 
> problem.Add queue scheduling deadLine in fairScheduler.When a queue is not 
> scheduled for FairScheduler within a specified time.We mandatory scheduler it!
>     On the basis of the above, I propose this issue...

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to