Min Shen created YARN-5734:

             Summary: OrgQueue for easy CapacityScheduler queue configuration 
                 Key: YARN-5734
                 URL: https://issues.apache.org/jira/browse/YARN-5734
             Project: Hadoop YARN
          Issue Type: New Feature
            Reporter: Min Shen
            Assignee: Min Shen

The current xml based configuration mechanism in CapacityScheduler makes it 
very inconvenient to apply any changes to the queue configurations. We saw 2 
main drawbacks in the file based configuration mechanism:
# This makes it very inconvenient to automate queue configuration updates. For 
example, in our cluster setup, we leverage the queue mapping feature from 
YARN-2411 to route users to their dedicated organization queues. It could be 
extremely cumbersome to keep updating the config file to manage the very 
dynamic mapping between users to organizations.
# Even a user has the admin permission on one specific queue, that user is 
unable to make any queue configuration changes to resize the subqueues, 
changing queue ACLs, or creating new queues. All these operations need to be 
performed in a centralized manner by the cluster administrators.

With these current limitations, we realized the need of a more flexible 
configuration mechanism that allows queue configurations to be stored and 
managed more dynamically. We developed the feature internally at LinkedIn which 
introduces the concept of MutableConfigurationProvider. What it essentially 
does is to provide a set of configuration mutation APIs that allows queue 
configurations to be updated externally with a set of REST APIs. When 
performing the queue configuration changes, the queue ACLs will be honored, 
which means only queue administrators can make configuration changes to a given 
queue. MutableConfigurationProvider is implemented as a pluggable interface, 
and we have one implementation of this interface which is based on Derby 
embedded database.

This feature has been deployed at LinkedIn's Hadoop cluster for a year now, and 
have gone through several iterations of gathering feedbacks from users and 
improving accordingly. With this feature, cluster administrators are able to 
automate lots of thequeue configuration management tasks, such as setting the 
queue capacities to adjust cluster resources between queues based on 
established resource consumption patterns, or managing updating the user to 
queue mappings. We have attached our design documentation with this ticket and 
would like to receive feedbacks from the community regarding how to best 
integrate it with the latest version of YARN.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to