[
https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16215881#comment-16215881
]
Wangda Tan commented on YARN-7117:
----------------------------------
Thanks [~suma.shivaprasad] for promptly working on the prototype. Reviewed and
had several long offline discussions with Suma,
*Some high level comments*:
1) The AutoCreatedParentQueue/LeafQueue are too similar to PlanQueue and
ReservationQueue, suggest to merge the two implementations.
2) It is a good idea to keep logics to do capacity management to a separate
module (which implements SchedulingEditPolicy). However currently
SchedulingEditPolicy needs to be preconfigured and cannot be turned on/off at
runtime. We need to improve that part before using SchedulingEditPolicy.
3) There're lots of common logics between ProportionalCapacityPreemptionPolicy
and the new added SchedulingEditPolicy, it's better to merge at least clone
queues logics / data structures and libraries to a common abstract class.
*Regarding to development, I think we can break down this JIRA to following sub
tasks:*
a. Rename ReservationQueue/PlanQueue to a different name (with as less as
possible logic changes), which can be used to implement this feature. (no
dependency)
b. Implement Queue Mapping / Queue Creation logics which covers: Get
applications queue mapping and auto create leaf queues. Reject application if
leaf queue creation failed. (depends on a.)
c. SchedulingEditPolicy changes to make it refreshable and can be turned on/off
while scheduler is running. (Which is tracked by YARN-7370), no dependency.
d. Move clone queues method / data structure / libaries from
ProportionalCapacityPreemptionPolicy to a common parent class. (no dependency)
e. Add SchedulingEditPolicy framework and implementation to adjust capacities
and states of sub queues. (depends on a/d)
*Regarding to development in branch v.s. development in trunk*:
Since a/c/d are all clean up / refactoring tasks not specific to this task, I
prefer to do them directly on trunk.
And after b, this feature is already end-to-end completed, I prefer to do b/e
on trunk as well to avoid overheads of feature branch.
[~curino] / [~jlowe] / [~subru], could you add your suggestions to the attached
prototype (See outlined workflow:
https://issues.apache.org/jira/secure/attachment/12893355/YARN-7117_Workflow.pdf)
and plans?
> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue
> Mapping
> ----------------------------------------------------------------------------------
>
> Key: YARN-7117
> URL: https://issues.apache.org/jira/browse/YARN-7117
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacity scheduler
> Reporter: Wangda Tan
> Assignee: Suma Shivaprasad
> Attachments:
> YARN-7117.Capacity.Scheduler.Support.Auto.Creation.Of.Leaf.Queue.pdf,
> YARN-7117.poc.1.patch, YARN-7117.poc.patch, YARN-7117_Workflow.pdf
>
>
> Currently Capacity Scheduler doesn't support auto creation of queues when
> doing queue mapping. We saw more and more use cases which has complex queue
> mapping policies configured to handle application to queues mapping.
> The most common use case of CapacityScheduler queue mapping is to create one
> queue for each user/group. However update {{capacity-scheduler.xml}} and
> {{RMAdmin:refreshQueues}} needs to be done when new user/group onboard. One
> of the option to solve the problem is automatically create queues when new
> user/group arrives.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
