[
https://issues.apache.org/jira/browse/YARN-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17656971#comment-17656971
]
ASF GitHub Bot commented on YARN-10965:
---------------------------------------
szilard-nemeth commented on code in PR #3470:
URL: https://github.com/apache/hadoop/pull/3470#discussion_r1066286419
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java:
##########
@@ -100,6 +101,7 @@ public CapacitySchedulerQueueManager(Configuration conf,
this.queueStateManager = new QueueStateManager<>();
this.appPriorityACLManager = appPriorityACLManager;
this.configuredNodeLabels = new ConfiguredNodeLabels();
+ this.queueCapacityHandler = new
CapacitySchedulerQueueCapacityHandler(labelManager);
Review Comment:
I can see that this field is only accessed by getQueueCapacityHandler which
is not used anywhere. Do we need to create the
CapacitySchedulerQueueCapacityHandler then?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueCapacityHandler.java:
##########
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.hadoop.yarn.api.records.Resource;
+import
org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueueCapacityVector.ResourceUnitCapacityType;
+import org.apache.hadoop.yarn.util.resource.ResourceCalculator;
+import org.apache.hadoop.yarn.util.resource.ResourceUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+
+import static
org.apache.hadoop.yarn.api.records.ResourceInformation.MEMORY_URI;
+import static
org.apache.hadoop.yarn.api.records.ResourceInformation.VCORES_URI;
+import static
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.NO_LABEL;
+
+/**
+ * Controls how capacity and resource values are set and calculated for a
queue.
+ * Effective minimum and maximum resource values are set for each label and
resource separately.
+ */
+public class CapacitySchedulerQueueCapacityHandler {
Review Comment:
I can see that this class is only used by tests. How can this happen? :)
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ResourceCalculationDriver.java:
##########
@@ -0,0 +1,368 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableSet;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueueCapacityVector.QueueCapacityVectorEntry;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueueCapacityVector.ResourceUnitCapacityType;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static
org.apache.hadoop.yarn.api.records.ResourceInformation.MEMORY_URI;
+
+/**
+ * Drives the main logic of resource calculation for all children under a
parent queue. Acts as a
+ * bookkeeper of disposable update information that is used by all children
under a common parent.
+ */
+public class ResourceCalculationDriver {
+ protected static final Set<ResourceUnitCapacityType> CALCULATOR_PRECEDENCE =
+ ImmutableSet.of(
+ ResourceUnitCapacityType.ABSOLUTE,
+ ResourceUnitCapacityType.PERCENTAGE,
+ ResourceUnitCapacityType.WEIGHT);
+
+ protected final QueueResourceRoundingStrategy roundingStrategy =
+ new DefaultQueueResourceRoundingStrategy(CALCULATOR_PRECEDENCE);
+ protected final CSQueue parent;
+ protected final QueueCapacityUpdateContext updateContext;
+ protected final Map<ResourceUnitCapacityType,
AbstractQueueCapacityCalculator> calculators;
+ protected final Collection<String> definedResources;
+
+ protected final Map<String, ResourceVector> overallRemainingResource = new
HashMap<>();
+ protected final Map<String, ResourceVector> batchRemainingResource = new
HashMap<>();
+ // Used by ABSOLUTE capacity types
+ protected final Map<String, ResourceVector> normalizedResourceRatio = new
HashMap<>();
+ // Used by WEIGHT capacity typet js
+ protected final Map<String, Map<String, Float>> sumWeightsPerLabel = new
HashMap<>();
+
+ protected String currentResourceName;
+ protected AbstractQueueCapacityCalculator currentCalculator;
+ protected CSQueue currentChild;
+ protected Map<String, Float> usedResourceByCurrentCalculator = new
HashMap<>();
+
+ public ResourceCalculationDriver(
+ CSQueue parent, QueueCapacityUpdateContext updateContext,
+ Map<ResourceUnitCapacityType, AbstractQueueCapacityCalculator>
calculators,
+ Collection<String> definedResources) {
+ this.parent = parent;
+ this.updateContext = updateContext;
+ this.calculators = calculators;
+ this.definedResources = definedResources;
+ }
+
+ /**
+ * Returns the parent that is driving the calculation.
+ *
+ * @return a common parent queue
+ */
+ public CSQueue getParent() {
+ return parent;
+ }
+
+ /**
+ * Returns the context that is used throughout the whole update phase.
+ *
+ * @return update context
+ */
+ public QueueCapacityUpdateContext getUpdateContext() {
+ return updateContext;
+ }
+
+ /**
+ * Returns the name of the resource that is currently processed.
+ *
+ * @return resource name
+ */
+ public String getCurrentResourceName() {
+ return currentResourceName;
+ }
+
+ /**
+ * Returns the child that is currently processed.
+ *
+ * @return child queue
+ */
+ public CSQueue getCurrentChild() {
+ return currentChild;
+ }
+
+ /**
+ * Sets the currently evaluated child to a specific queue.
+ *
+ * @param currentChild a child queue
+ */
+ public void setCurrentChild(CSQueue currentChild) {
+ if (currentChild.getParent() != parent) {
+ throw new IllegalArgumentException("Child queue " +
currentChild.getQueuePath() + " is not " +
+ "a child of " + parent.getQueuePath());
+ }
+
+ this.currentChild = currentChild;
+ }
+
+ /**
+ * A shorthand to return the minimum capacity vector entry for the currently
evaluated child and
+ * resource name.
+ *
+ * @param label node label
+ * @return capacity vector entry
+ */
+ public QueueCapacityVectorEntry getCurrentMinimumCapacityEntry(String label)
{
+ return
currentChild.getConfiguredCapacityVector(label).getResource(currentResourceName);
+ }
+
+ /**
+ * A shorthand to return the maximum capacity vector entry for the currently
evaluated child and
+ * resource name.
+ *
+ * @param label node label
+ * @return capacity vector entry
+ */
+ public QueueCapacityVectorEntry getCurrentMaximumCapacityEntry(String label)
{
+ return
currentChild.getConfiguredMaxCapacityVector(label).getResource(currentResourceName);
+ }
+
+ /**
+ * Increments the aggregated weight.
+ *
+ * @param label node label
+ * @param resourceName resource unit name
+ * @param value weight value
+ */
+ public void incrementWeight(String label, String resourceName, float value) {
+ sumWeightsPerLabel.putIfAbsent(label, new HashMap<>());
+ sumWeightsPerLabel.get(label).put(resourceName,
+ sumWeightsPerLabel.get(label).getOrDefault(resourceName, 0f) + value);
Review Comment:
@9uapaw Is this comment fixed?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueCapacityHandler.java:
##########
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.hadoop.yarn.api.records.Resource;
+import
org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueueCapacityVector.ResourceUnitCapacityType;
+import org.apache.hadoop.yarn.util.resource.ResourceCalculator;
+import org.apache.hadoop.yarn.util.resource.ResourceUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+
+import static
org.apache.hadoop.yarn.api.records.ResourceInformation.MEMORY_URI;
+import static
org.apache.hadoop.yarn.api.records.ResourceInformation.VCORES_URI;
+import static
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.NO_LABEL;
+
+/**
+ * Controls how capacity and resource values are set and calculated for a
queue.
+ * Effective minimum and maximum resource values are set for each label and
resource separately.
+ */
+public class CapacitySchedulerQueueCapacityHandler {
Review Comment:
Never mind, I can see the
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbsoluteResourceCapacityCalculator#updateCapacitiesAfterCalculation
method.
But, only the method called
"CapacitySchedulerQueueCapacityHandler.setQueueCapacities" is called, all other
public methods are not which is still a bit strange.
> Centralize queue resource calculation based on CapacityVectors
> --------------------------------------------------------------
>
> Key: YARN-10965
> URL: https://issues.apache.org/jira/browse/YARN-10965
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Andras Gyori
> Assignee: Andras Gyori
> Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> With the introduction of YARN-10930 it is possible to unify queue resource
> calculation. In order to narrow down the scope of this patch, the base system
> is implemented here, without refactoring the existing resource calculation in
> updateClusterResource (which will be done in YARN-11000).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
