Jonathan Hung commented on YARN-5734:

[~curino], thanks for the comments.

For 1 and 2, this is in our plans (to do either internally or e.g. in a feature 
branch). The Derby based storage is one implementation (and eventually we will 
implement an RMStateStore version). 

I took a quick look at some of the ReservationSystem code - my understanding is 
that the {{PlanQueue}}'s capacity/max-capacity is currently mutable in the same 
way as {{ParentQueue}} (i.e. via {{refreshQueues}})? The dynamic part is in the 
{{ReservationQueue}}. So instead of having to {{setEntitlement}} for each child 
of a {{PlanQueue}}, we can leverage the MutableConfigurationProvider API to 
change all child queue capacities of a {{PlanQueue}}. Is this what you had in 
mind? Also changing queue configurations such as user-limit or 
user-limit-factor of a {{ReservationQueue}} can be done via this API (as can 
other configurations if they are added to {{ReservationQueue}} in the future). 

> OrgQueue for easy CapacityScheduler queue configuration management
> ------------------------------------------------------------------
>                 Key: YARN-5734
>                 URL: https://issues.apache.org/jira/browse/YARN-5734
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Min Shen
>            Assignee: Min Shen
>         Attachments: OrgQueue_Design_v0.pdf
> The current xml based configuration mechanism in CapacityScheduler makes it 
> very inconvenient to apply any changes to the queue configurations. We saw 2 
> main drawbacks in the file based configuration mechanism:
> # This makes it very inconvenient to automate queue configuration updates. 
> For example, in our cluster setup, we leverage the queue mapping feature from 
> YARN-2411 to route users to their dedicated organization queues. It could be 
> extremely cumbersome to keep updating the config file to manage the very 
> dynamic mapping between users to organizations.
> # Even a user has the admin permission on one specific queue, that user is 
> unable to make any queue configuration changes to resize the subqueues, 
> changing queue ACLs, or creating new queues. All these operations need to be 
> performed in a centralized manner by the cluster administrators.
> With these current limitations, we realized the need of a more flexible 
> configuration mechanism that allows queue configurations to be stored and 
> managed more dynamically. We developed the feature internally at LinkedIn 
> which introduces the concept of MutableConfigurationProvider. What it 
> essentially does is to provide a set of configuration mutation APIs that 
> allows queue configurations to be updated externally with a set of REST APIs. 
> When performing the queue configuration changes, the queue ACLs will be 
> honored, which means only queue administrators can make configuration changes 
> to a given queue. MutableConfigurationProvider is implemented as a pluggable 
> interface, and we have one implementation of this interface which is based on 
> Derby embedded database.
> This feature has been deployed at LinkedIn's Hadoop cluster for a year now, 
> and have gone through several iterations of gathering feedbacks from users 
> and improving accordingly. With this feature, cluster administrators are able 
> to automate lots of thequeue configuration management tasks, such as setting 
> the queue capacities to adjust cluster resources between queues based on 
> established resource consumption patterns, or managing updating the user to 
> queue mappings. We have attached our design documentation with this ticket 
> and would like to receive feedbacks from the community regarding how to best 
> integrate it with the latest version of YARN.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to