[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985160#comment-13985160
]
Sunil G commented on YARN-1963:
-------------------------------
We have done few analysis and implemented support for application priority.
I wish to share the thoughts here, kindly check the same.
Design thoughts:
1. Configuration Part
We planned to use some existing priority configuration as given below. These
are used to set a Job priority.
a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority)
b. We can also use configuration "mapreduce.job.priority".
The values for priority can be VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW
2. Scheduler Side
If the Capacity Scheduler queue has multiple applications(Jobs) to run with
different priorities, CS will allocate containers for the highest priority
application and then for next priority and so on.
When multiple queues are configured with different capacities, this priority
will work internal to the each queue.
For this, we planned to add a priority comparison check in the below data
structure.
Comparator<FiCaSchedulerApp> applicationComparator
We added a priority check here in compare() of applicationComparator while
selecting applications. Updated design here will be like,
1. Check for priority first. If there, return highest priority job.
2. Continue existing logic such as App ID comparison and TimeStamp
comparison.
With these changes, we can make highest priority job will get preference in a
queue.
NB: In addition to this, we added a preemption module also to get High priority
jobs resources fast by preempting lower priority ones.
I wish to upload a patch if this approach is fine.
> Support priorities across applications within the same queue
> -------------------------------------------------------------
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
