[
https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277903#comment-16277903
]
Wangda Tan commented on YARN-7473:
----------------------------------
Thanks [~suma.shivaprasad], I took a look at some of updated part (especially
for the Configuration passed to LeafQueue code paths). Some comments so far:
1) AbstractManagedParentQueue:
{code}
CapacitySchedulerConfiguration leafQueueConfigs = new
CapacitySchedulerConfiguration(new Configuration(false));
{code}
Should be
{code}
CapacitySchedulerConfiguration leafQueueConfigs = new
CapacitySchedulerConfiguration(new Configuration(false), false);
{code}
And
{code}
// .getConfiguration().getAllPropertiesByTag(YarnPropertyTag
// .RESOURCEMANAGER).iterator()
{code}
Should be removed.
2) setEntitlement should be removed from ReservationQueue
3)
{code}
public void setEntitlement(String nodeLabel, QueueEntitlement entitlement)
{code}
This one should be removed. And see my next comment.
4) Why call updateCapacitiesToZero inside AutoCreatedLeafQueue#initialize? Will
this cause jittering of capacity? (When reinitialize happens, all queue’s
capacities will be set to 0 and soon updated by CapacityManager?
5) Inside LeafQueue#setupQueueConfigs, now all methods are get from the
Configuration passed-in. However, I think we should only fetch queue-specific
configs from the passed-in Configuration reference. And for global configs such
as
{code}
nodeLocalityDelay = conf.getNodeLocalityDelay();
rackLocalityAdditionalDelay = conf.getRackLocalityAdditionalDelay();
rackLocalityFullReset = conf.getRackLocalityFullReset();
{code}
Should get from this.configuration.
We may need to add a test for this.
> Implement Framework and policy for capacity management of auto created queues
> ------------------------------------------------------------------------------
>
> Key: YARN-7473
> URL: https://issues.apache.org/jira/browse/YARN-7473
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Suma Shivaprasad
> Assignee: Suma Shivaprasad
> Attachments: YARN-7473.1.patch, YARN-7473.10.patch,
> YARN-7473.11.patch, YARN-7473.12.patch, YARN-7473.2.patch, YARN-7473.3.patch,
> YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch, YARN-7473.7.patch,
> YARN-7473.8.patch, YARN-7473.9.patch
>
>
> This jira mainly addresses the following
>
> 1.Support adding pluggable policies on parent queue for dynamically managing
> capacity/state for leaf queues.
> 2. Implement a default policy that manages capacity based on pending
> applications and either grants guaranteed or zero capacity to queues based on
> parent's available guaranteed capacity.
> 3. Integrate with SchedulingEditPolicy framework to trigger this periodically
> and signal scheduler to take necessary actions for capacity/queue management.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
