[jira] [Commented] (YARN-2257) Add user to queue mappings to automatically place users' apps into specific queues
[
https://issues.apache.org/jira/browse/YARN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14057339#comment-14057339
]
Patrick Liu commented on YARN-2257:
---
Hi, Vinod,
I think we could inject the user->queue mapping judgement in 'RMAppManager's
method 'protected synchronized void submitApplication':
// Sanity checks
if (submissionContext.getQueue() == null) {
submissionContext.setQueue(YarnConfiguration.DEFAULT_QUEUE_NAME);
}
if (submissionContext.getApplicationName() == null) {
submissionContext.setApplicationName(
YarnConfiguration.DEFAULT_APPLICATION_NAME);
}
All applications submitted to yarn will be launched by 'RMAppManager'.
'RMAppManager' will do sanity check, create a 'RMAppImpl' instance, and finally
send the 'new RMAppEvent(applicationId, RMAppEventType.START)' event.
When 'RMAppImpl' received the Event, it will change the state machine and do
the transition. The transition will launch the 'RMAppAttemptImpl', and start
the 'RMAppAttemptImpl'.
Then the app will be scheduled by the specific scheduler.
The only thing we need to injuect is the QUEUE in the submissionContext.
Like this:
// Precondition: set "user-as-default-queue" to false in yarn-site.xml
if(QueuePlacementRule.hasMappingForUser(user)) {
submissionContext.setQueue(QueuePlacementRule.getQueue(user));
} else {
submissionContext.setQueue(Default);
}
> Add user to queue mappings to automatically place users' apps into specific
> queues
> --
>
> Key: YARN-2257
> URL: https://issues.apache.org/jira/browse/YARN-2257
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Reporter: Patrick Liu
>Assignee: Vinod Kumar Vavilapalli
> Labels: features
>
> Currently, the fair-scheduler supports two modes, default queue or individual
> queue for each user.
> Apparently, the default queue is not a good option, because the resources
> cannot be managed for each user or group.
> However, individual queue for each user is not good enough. Especially when
> connecting yarn with hive. There will be increasing hive users in a corporate
> environment. If we create a queue for a user, the resource management will be
> hard to maintain.
> I think the problem can be solved like this:
> 1. Define user->queue mapping in Fair-Scheduler.xml. Inside each queue, use
> aclSubmitApps to control user's ability.
> 2. Each time a user submit an app to yarn, if the user has mapped to a queue,
> the app will be scheduled to that queue; otherwise, the app will be submitted
> to default queue.
> 3. If the user cannot pass aclSubmitApps limits, the app will not be accepted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2257) Add user to queue mappings to automatically place users' apps into specific queues
[ https://issues.apache.org/jira/browse/YARN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054818#comment-14054818 ] Patrick Liu commented on YARN-2257: --- Hi, [~vinodkv]. Although I am new, I want challenges. And I will follow this thread. If I am capable of handling it, I will try. I am using hadoop2.0.0-cdh4.6.0. If I want to fix this issue, which version should I download and try fix? If we want to add the user->queue mapping ability in fair-scheduler, I think we should enhance the QueueManagerInfo, and add a map containing user->queue mapping in QueueManager. The mapping will only affect the process when a new application submitted, so the 'handle' method in FairScheduler will be modified. We can add a new method when 'APP_ADDED' occurred, and dispatch the app to the given queue defined in user->queue map. However as [~kkambatl] suggested, we should transfer QueuePlacementRule feature to yarn. If so, how many existing features will be affected? > Add user to queue mappings to automatically place users' apps into specific > queues > -- > > Key: YARN-2257 > URL: https://issues.apache.org/jira/browse/YARN-2257 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Patrick Liu >Assignee: Vinod Kumar Vavilapalli > Labels: features > > Currently, the fair-scheduler supports two modes, default queue or individual > queue for each user. > Apparently, the default queue is not a good option, because the resources > cannot be managed for each user or group. > However, individual queue for each user is not good enough. Especially when > connecting yarn with hive. There will be increasing hive users in a corporate > environment. If we create a queue for a user, the resource management will be > hard to maintain. > I think the problem can be solved like this: > 1. Define user->queue mapping in Fair-Scheduler.xml. Inside each queue, use > aclSubmitApps to control user's ability. > 2. Each time a user submit an app to yarn, if the user has mapped to a queue, > the app will be scheduled to that queue; otherwise, the app will be submitted > to default queue. > 3. If the user cannot pass aclSubmitApps limits, the app will not be accepted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2257) Add user to queue mappings to automatically place users' apps into specific queues
[ https://issues.apache.org/jira/browse/YARN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054683#comment-14054683 ] Patrick Liu commented on YARN-2257: --- Thanks for all of your consideration, cause this is my first report. Agree with both Sandy and Vinod, it is better to add QueuePlacementRule to general framework rather than specific scheduler. Besides, I think the user-specific default is a good suggestion. Currently, I use an alias like this "alias hive='hive -hiveconf mapreduce.job.queuename=queue_abc" to start hive. With user-specific default queue, this effort can be saved. And it makes configurations more maintainable. It's difficult for me to make the patch since I just started reading hadoop code. [~vinodkv] please do it. Many thanks. > Add user to queue mappings to automatically place users' apps into specific > queues > -- > > Key: YARN-2257 > URL: https://issues.apache.org/jira/browse/YARN-2257 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Patrick Liu >Assignee: Vinod Kumar Vavilapalli > Labels: features > > Currently, the fair-scheduler supports two modes, default queue or individual > queue for each user. > Apparently, the default queue is not a good option, because the resources > cannot be managed for each user or group. > However, individual queue for each user is not good enough. Especially when > connecting yarn with hive. There will be increasing hive users in a corporate > environment. If we create a queue for a user, the resource management will be > hard to maintain. > I think the problem can be solved like this: > 1. Define user->queue mapping in Fair-Scheduler.xml. Inside each queue, use > aclSubmitApps to control user's ability. > 2. Each time a user submit an app to yarn, if the user has mapped to a queue, > the app will be scheduled to that queue; otherwise, the app will be submitted > to default queue. > 3. If the user cannot pass aclSubmitApps limits, the app will not be accepted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-2257) Add user to queue mapping in Fair-Scheduler
Patrick Liu created YARN-2257: - Summary: Add user to queue mapping in Fair-Scheduler Key: YARN-2257 URL: https://issues.apache.org/jira/browse/YARN-2257 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Reporter: Patrick Liu Currently, the fair-scheduler supports two modes, default queue or individual queue for each user. Apparently, the default queue is not a good option, because the resources cannot be managed for each user or group. However, individual queue for each user is not good enough. Especially when connecting yarn with hive. There will be increasing hive users in a corporate environment. If we create a queue for a user, the resource management will be hard to maintain. I think the problem can be solved like this: 1. Define user->queue mapping in Fair-Scheduler.xml. Inside each queue, use aclSubmitApps to control user's ability. 2. Each time a user submit an app to yarn, if the user has mapped to a queue, the app will be scheduled to that queue; otherwise, the app will be submitted to default queue. 3. If the user cannot pass aclSubmitApps limits, the app will not be accepted. -- This message was sent by Atlassian JIRA (v6.2#6252)
