[
https://issues.apache.org/jira/browse/YARN-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14108932#comment-14108932
]
Wangda Tan commented on YARN-1707:
----------------------------------
Hi [~curino],
Thanks for updating, I just took a look, some minor comments,
1) CapacityScheduler#removeQueue
{code}
if (disposableLeafQueue.getCapacity() > 0) {
throw new SchedulerConfigEditException("The queue " + queueName
+ " has non-zero capacity: " + disposableLeafQueue.getCapacity());
}
{code}
removeQueue check disposableLeafQueue's capacity > 0, but addQueue doesn't
check. In addition,
After previous check, ParentQueue#removeChildQueue/addChildQueue doesn't need
check its capacity again.
And they should throw same type of exception (both SchedulerConfigEditException
or both IllegalArgumentException)
2) CS#addQueue
{code}
throw new SchedulerConfigEditException("Queue " + queue.getQueueName()
+ " is not a dynamic Queue");
{code}
Should "dynamic Queue" should be "reservation queue" comparing to similar
exception throw in removeQueue?
3) CS#setEntitlement
{code}
if (sesConf.getCapacity() > queue.getCapacity()) {
newQueue.addCapacity((sesConf.getCapacity() - queue.getCapacity()));
} else {
newQueue
.subtractCapacity((queue.getCapacity() - sesConf.getCapacity()));
}
{code}
Maybe it's better to merge the add/substractCapacity to changeCapacity(delta)
Or just create a "setCapacity" in ReservationQueue?
4) CS#getReservableQueues
Is it better to rename it to getPlanQueues?
5) ReservationQueue#getQueueName
{code}
@Override
public String getQueueName() {
return this.getParent().getQueueName();
}
{code}
I'm not sure why doing this, could you please elaborate? This makes
this.queueName and this.getQueueName has different semantic.
6) ReservationQueue#substractCapacity
{code}
this.setCapacity(this.getCapacity() - capacity);
{code}
With EPSILON, it is possible this.capacity < 0 set substract, its better to cap
this.capacity in range of [0,1]. Also addCapacity
7) DynamicQueueConf
I think unfold it to two float as parameter for setEntitlement maybe more
straigtforward, is it possible more fields will be add to DynamicQueueConf?
8) ParentQueue#setChildQueues
Since only PlanQueue need sum of capacity <= 1, I would suggest make this
method protected, and PlanQueue can overwrite this method. Or add a check in
ParentQueue#setChildQueues.
Wangda
> Making the CapacityScheduler more dynamic
> -----------------------------------------
>
> Key: YARN-1707
> URL: https://issues.apache.org/jira/browse/YARN-1707
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Labels: capacity-scheduler
> Attachments: YARN-1707.2.patch, YARN-1707.3.patch, YARN-1707.patch
>
>
> The CapacityScheduler is a rather static at the moment, and refreshqueue
> provides a rather heavy-handed way to reconfigure it. Moving towards
> long-running services (tracked in YARN-896) and to enable more advanced
> admission control and resource parcelling we need to make the
> CapacityScheduler more dynamic. This is instrumental to the umbrella jira
> YARN-1051.
> Concretely this require the following changes:
> * create queues dynamically
> * destroy queues dynamically
> * dynamically change queue parameters (e.g., capacity)
> * modify refreshqueue validation to enforce sum(child.getCapacity())<= 100%
> instead of ==100%
> We limit this to LeafQueues.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
