[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262562#comment-17262562
]
zhuqi edited comment on YARN-10504 at 1/11/21, 10:42 AM:
---------------------------------------------------------
[~wangda] [~bteke]
When used the absolute resource in the autoCreatedLeafQueue, the logic is :
1.Initialize the update absolute template:
{code:java}
protected AutoCreatedLeafQueueConfig.Builder initializeLeafQueueConfigs()
throws IOException {
....
//Update the absolute template
if (this.capacityConfigType.equals(CapacityConfigType.ABSOLUTE_RESOURCE)) {
updateQueueCapacities(queueCapacities);
}
...
builder.capacities(queueCapacities);
return builder;
}
//get resource value in
private Resource internalGetLabeledResourceRequirementForQueue(String queue,
String label, Set<String> resourceTypes, String suffix) {
...
if (matcher.find()) {
// Get the sub-group.
String subGroup = matcher.group(0);
if (subGroup.trim().isEmpty()) {
return Resources.none();
}
subGroup = subGroup.substring(1, subGroup.length() - 1);
for (String kvPair : subGroup.trim().split(",")) {
String[] splits = kvPair.split("=");
// Ensure that each sub string is key value pair separated by '='.
if (splits != null && splits.length > 1) {
updateResourceValuesFromConfig(resourceTypes, resource, splits);
}
}
}
...
return resource;
}
// Update in ManagedParentQueue's updateQueueCapacities
private void updateQueueCapacities(QueueCapacities queueCapacities) {
for (String label : queueCapacities.getExistingNodeLabels()) {
queueCapacities.setCapacity(label,
this.csContext.getResourceCalculator().divide(
this.csContext.getClusterResource(),
this.csContext.getConfiguration().getMinimumResourceRequirement(
label,
this.csContext.getConfiguration()
.getAutoCreatedQueueTemplateConfPrefix(getQueuePath()),
resourceTypes),
getQueueResourceQuotas().getConfiguredMinResource(label)));
Resource childMaxResource = this.csContext.getConfiguration()
.getMaximumResourceRequirement(label,
this.csContext.getConfiguration()
.getAutoCreatedQueueTemplateConfPrefix(getQueuePath()),
resourceTypes);
Resource parentMaxRes = getQueueResourceQuotas()
.getConfiguredMaxResource(label);
Resource effMaxResource = Resources.min(
this.csContext.getResourceCalculator(),
this.csContext.getClusterResource(),
childMaxResource.equals(Resources.none()) ? parentMaxRes
: childMaxResource,
parentMaxRes);
queueCapacities.setMaximumCapacity(
label, this.csContext.getResourceCalculator().divide(
this.csContext.getClusterResource(),
effMaxResource,
getQueueResourceQuotas().getConfiguredMaxResource(label)));
queueCapacities.setAbsoluteCapacity(
label, queueCapacities.getCapacity(label)
* getQueueCapacities().getAbsoluteCapacity(label));
queueCapacities.setAbsoluteMaximumCapacity(label,
queueCapacities.getMaximumCapacity(label)
* getQueueCapacities().getAbsoluteMaximumCapacity(label));
}
} {code}
2. Now, the capacity has been updated to absolute resource based value in
addChildQueue:
It back to the absolute resource in : setEffectiveMinResource
{code:java}
public void mergeCapacities(QueueCapacities capacities) {
for ( String nodeLabel : capacities.getExistingNodeLabels()) {
queueCapacities.setCapacity(nodeLabel,
capacities.getCapacity(nodeLabel));
queueCapacities.setAbsoluteCapacity(nodeLabel, capacities
.getAbsoluteCapacity(nodeLabel));
queueCapacities.setMaximumCapacity(nodeLabel, capacities
.getMaximumCapacity(nodeLabel));
queueCapacities.setAbsoluteMaximumCapacity(nodeLabel, capacities
.getAbsoluteMaximumCapacity(nodeLabel));
Resource resourceByLabel = labelManager.getResourceByLabel(nodeLabel,
csContext.getClusterResource());
getQueueResourceQuotas().setEffectiveMinResource(nodeLabel,
Resources.multiply(resourceByLabel,
queueCapacities.getAbsoluteCapacity(nodeLabel)));
getQueueResourceQuotas().setEffectiveMaxResource(nodeLabel,
Resources.multiply(resourceByLabel, queueCapacities
.getAbsoluteMaximumCapacity(nodeLabel)));
}
}{code}
The effective resource have been updated to autoCreatedLeafQueue already.
And now, the result is consistent with origin test case, because also use
Resources.multiply(resourceByLabel,
queueCapacities.getAbsoluteCapacity(nodeLabel))
to get result.
3. Then, in LeafQueue's updateClusterResource:
{code:java}
public void updateClusterResource(Resource clusterResource,
ResourceLimits currentResourceLimits) {
writeLock.lock();
try {
...
super.updateEffectiveResources(clusterResource);
...}
}{code}
We can change the updateEffectiveResources , to get consistent result to
origin:
{code:java}
void updateEffectiveResources(Resource clusterResource) {
Set<String> configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
Resource resourceByLabel = labelManager.getResourceByLabel(label,
clusterResource);
Resource minResource = queueResourceQuotas.getConfiguredMinResource(
label);
// Update effective resource (min/max) to each child queue.
if (getCapacityConfigType().equals(
...
//The round loss caused by here
CapacityConfigType.ABSOLUTE_RESOURCE) && !(this instanceof
AutoCreatedLeafQueue)) {
...
} else{
queueResourceQuotas.setEffectiveMinResource(label, Resources
.multiply(resourceByLabel,
queueCapacities.getAbsoluteCapacity(label)));
queueResourceQuotas.setEffectiveMaxResource(label, Resources
.multiply(resourceByLabel,
queueCapacities.getAbsoluteMaximumCapacity(label)));
}
}
}
{code}
If you any advice for this.
Thanks.
was (Author: zhuqi):
[~wangda] [~bteke]
When used the absolute resource in the autoCreatedLeafQueue, the logic is :
1.Initialize the update absolute template:
{code:java}
protected AutoCreatedLeafQueueConfig.Builder initializeLeafQueueConfigs()
throws IOException {
....
//Update the absolute template
if (this.capacityConfigType.equals(CapacityConfigType.ABSOLUTE_RESOURCE)) {
updateQueueCapacities(queueCapacities);
}
...
builder.capacities(queueCapacities);
return builder;
}
//get resource value in
private Resource internalGetLabeledResourceRequirementForQueue(String queue,
String label, Set<String> resourceTypes, String suffix) {
...
if (matcher.find()) {
// Get the sub-group.
String subGroup = matcher.group(0);
if (subGroup.trim().isEmpty()) {
return Resources.none();
}
subGroup = subGroup.substring(1, subGroup.length() - 1);
for (String kvPair : subGroup.trim().split(",")) {
String[] splits = kvPair.split("=");
// Ensure that each sub string is key value pair separated by '='.
if (splits != null && splits.length > 1) {
updateResourceValuesFromConfig(resourceTypes, resource, splits);
}
}
}
...
return resource;
}
// Update in ManagedParentQueue's updateQueueCapacities
private void updateQueueCapacities(QueueCapacities queueCapacities) {
for (String label : queueCapacities.getExistingNodeLabels()) {
queueCapacities.setCapacity(label,
this.csContext.getResourceCalculator().divide(
this.csContext.getClusterResource(),
this.csContext.getConfiguration().getMinimumResourceRequirement(
label,
this.csContext.getConfiguration()
.getAutoCreatedQueueTemplateConfPrefix(getQueuePath()),
resourceTypes),
getQueueResourceQuotas().getConfiguredMinResource(label)));
Resource childMaxResource = this.csContext.getConfiguration()
.getMaximumResourceRequirement(label,
this.csContext.getConfiguration()
.getAutoCreatedQueueTemplateConfPrefix(getQueuePath()),
resourceTypes);
Resource parentMaxRes = getQueueResourceQuotas()
.getConfiguredMaxResource(label);
Resource effMaxResource = Resources.min(
this.csContext.getResourceCalculator(),
this.csContext.getClusterResource(),
childMaxResource.equals(Resources.none()) ? parentMaxRes
: childMaxResource,
parentMaxRes);
queueCapacities.setMaximumCapacity(
label, this.csContext.getResourceCalculator().divide(
this.csContext.getClusterResource(),
effMaxResource,
getQueueResourceQuotas().getConfiguredMaxResource(label)));
queueCapacities.setAbsoluteCapacity(
label, queueCapacities.getCapacity(label)
* getQueueCapacities().getAbsoluteCapacity(label));
queueCapacities.setAbsoluteMaximumCapacity(label,
queueCapacities.getMaximumCapacity(label)
* getQueueCapacities().getAbsoluteMaximumCapacity(label));
}
} {code}
2. Now, the capacity has been updated to absolute resource based value in
addChildQueue:
It back to the absolute resource in : setEffectiveMinResource
{code:java}
public void mergeCapacities(QueueCapacities capacities) {
for ( String nodeLabel : capacities.getExistingNodeLabels()) {
queueCapacities.setCapacity(nodeLabel,
capacities.getCapacity(nodeLabel));
queueCapacities.setAbsoluteCapacity(nodeLabel, capacities
.getAbsoluteCapacity(nodeLabel));
queueCapacities.setMaximumCapacity(nodeLabel, capacities
.getMaximumCapacity(nodeLabel));
queueCapacities.setAbsoluteMaximumCapacity(nodeLabel, capacities
.getAbsoluteMaximumCapacity(nodeLabel));
Resource resourceByLabel = labelManager.getResourceByLabel(nodeLabel,
csContext.getClusterResource());
getQueueResourceQuotas().setEffectiveMinResource(nodeLabel,
Resources.multiply(resourceByLabel,
queueCapacities.getAbsoluteCapacity(nodeLabel)));
getQueueResourceQuotas().setEffectiveMaxResource(nodeLabel,
Resources.multiply(resourceByLabel, queueCapacities
.getAbsoluteMaximumCapacity(nodeLabel)));
}
}{code}
The effective resource have been updated to autoCreatedLeafQueue already.
3. Then, in LeafQueue's updateClusterResource:
{code:java}
public void updateClusterResource(Resource clusterResource,
ResourceLimits currentResourceLimits) {
writeLock.lock();
try {
...
super.updateEffectiveResources(clusterResource);
...}
}{code}
We can change the updateEffectiveResources , to get consistent result to orgin:
{code:java}
void updateEffectiveResources(Resource clusterResource) {
Set<String> configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
Resource resourceByLabel = labelManager.getResourceByLabel(label,
clusterResource);
Resource minResource = queueResourceQuotas.getConfiguredMinResource(
label);
// Update effective resource (min/max) to each child queue.
if (getCapacityConfigType().equals(
...
//The round loss caused by here
CapacityConfigType.ABSOLUTE_RESOURCE) && !(this instanceof
AutoCreatedLeafQueue)) {
...
} else{
queueResourceQuotas.setEffectiveMinResource(label, Resources
.multiply(resourceByLabel,
queueCapacities.getAbsoluteCapacity(label)));
queueResourceQuotas.setEffectiveMaxResource(label, Resources
.multiply(resourceByLabel,
queueCapacities.getAbsoluteMaximumCapacity(label)));
}
}
}
{code}
If you any advice for this.
Thanks.
> Implement weight mode in Capacity Scheduler
> -------------------------------------------
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Benjamin Teke
> Assignee: Benjamin Teke
> Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
