Maysam Yabandeh commented on YARN-1963:

Thanks [~sunilg] for the design doc.

It might be useful if I share with you our use cases. 

Our most important use case is to let the admin change an app priority at 
runtime while it is running. The example is when a job gets unlucky taking much 
longer than usual due to some node failures or bugs. The user complains that 
the job is about to miss the deadline and admin needs a way to prioritize the 
user's job over the other jobs in the queue. This use case seems to be 
mentioned in Item 3 of Section 1.5.3 in the design doc but its "priority" seems 
not to be high.

Another use case is to dynamically give a job higher priority based on the job 
status. For example, when mapper fails and there is no headroom in the queue, 
the job preempt its reducers to make space for its mappers. The freed space is 
however not necessarily offered back to the job in fair scheduling. Ideally job 
could increase its priority when its reducers are being stalled waiting for its 
mappers to be assigned.

bq. Once all these requests of higher priority applications are served, then 
lower priority application requests will get served from Resource Manager.

We are using fair scheduler and I assumed this jira is to also cover that since 
YARN-2098 created as a sub-task. The design doc however seems to be fairly 
centered around CapacityScheduler. In the case of fair scheduler, I guess the 
priority can also be incorporated to the fair share calculation, instead of the 
strict order of high priority first.

> Support priorities across applications within the same queue 
> -------------------------------------------------------------
>                 Key: YARN-1963
>                 URL: https://issues.apache.org/jira/browse/YARN-1963
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: api, resourcemanager
>            Reporter: Arun C Murthy
>            Assignee: Sunil G
>         Attachments: YARN Application Priorities Design.pdf
> It will be very useful to support priorities among applications within the 
> same queue, particularly in production scenarios. It allows for finer-grained 
> controls without having to force admins to create a multitude of queues, plus 
> allows existing applications to continue using existing queues which are 
> usually part of institutional memory.

This message was sent by Atlassian JIRA

Reply via email to