[
https://issues.apache.org/jira/browse/YARN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054617#comment-14054617
]
Sandy Ryza commented on YARN-2257:
----------------------------------
To add some background: since MR1, the Fair Scheduler has been able to place
apps into queues named with the username or group of the submitter. Last year,
YARN-1392 extended this to accept more general policies - essentially any
function of (submitter's username, submitter's groups, requested queue), with a
structure that allows phrasing the policy in terms of simple rules and
fallbacks.
This generality is useful because different organizations have a variety of
ways they organize their users on their Hadoop clusters. Some administrators
want to be able to decide which queue a user's job goes into by placing them
into a unix group, while others want a queue for every user, with the option
for certain users to intentionally submit their jobs to certain queues. The
Fair Scheduler also in particular has need for some added complexity here,
because it models users within a queue with their own queues, unlike the
Capacity Scheduler, which has a construct for this.
We chose to put these queue placement policies in the Fair Scheduler because
other schedulers didn't have a precedent for placing apps in queues other than
the one requested, but my opinion is that these they could be a useful feature
for YARN. If not, we should at least add user->queue mappings in a way that's
compatible with more general mappings.
> Add user to queue mappings to automatically place users' apps into specific
> queues
> ----------------------------------------------------------------------------------
>
> Key: YARN-2257
> URL: https://issues.apache.org/jira/browse/YARN-2257
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
> Reporter: Patrick Liu
> Assignee: Vinod Kumar Vavilapalli
> Labels: features
>
> Currently, the fair-scheduler supports two modes, default queue or individual
> queue for each user.
> Apparently, the default queue is not a good option, because the resources
> cannot be managed for each user or group.
> However, individual queue for each user is not good enough. Especially when
> connecting yarn with hive. There will be increasing hive users in a corporate
> environment. If we create a queue for a user, the resource management will be
> hard to maintain.
> I think the problem can be solved like this:
> 1. Define user->queue mapping in Fair-Scheduler.xml. Inside each queue, use
> aclSubmitApps to control user's ability.
> 2. Each time a user submit an app to yarn, if the user has mapped to a queue,
> the app will be scheduled to that queue; otherwise, the app will be submitted
> to default queue.
> 3. If the user cannot pass aclSubmitApps limits, the app will not be accepted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
