[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148474#comment-14148474
]
Maysam Yabandeh commented on YARN-1963:
---------------------------------------
Thanks [~sunilg] for the design doc.
It might be useful if I share with you our use cases.
Our most important use case is to let the admin change an app priority at
runtime while it is running. The example is when a job gets unlucky taking much
longer than usual due to some node failures or bugs. The user complains that
the job is about to miss the deadline and admin needs a way to prioritize the
user's job over the other jobs in the queue. This use case seems to be
mentioned in Item 3 of Section 1.5.3 in the design doc but its "priority" seems
not to be high.
Another use case is to dynamically give a job higher priority based on the job
status. For example, when mapper fails and there is no headroom in the queue,
the job preempt its reducers to make space for its mappers. The freed space is
however not necessarily offered back to the job in fair scheduling. Ideally job
could increase its priority when its reducers are being stalled waiting for its
mappers to be assigned.
bq. Once all these requests of higher priority applications are served, then
lower priority application requests will get served from Resource Manager.
We are using fair scheduler and I assumed this jira is to also cover that since
YARN-2098 created as a sub-task. The design doc however seems to be fairly
centered around CapacityScheduler. In the case of fair scheduler, I guess the
priority can also be incorporated to the fair share calculation, instead of the
strict order of high priority first.
> Support priorities across applications within the same queue
> -------------------------------------------------------------
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
> Reporter: Arun C Murthy
> Assignee: Sunil G
> Attachments: YARN Application Priorities Design.pdf
>
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
