[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135350#comment-16135350
]
ASF GitHub Bot commented on YARN-2162:
--------------------------------------
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134269384
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
---
@@ -287,27 +289,69 @@ public float getReservableNodes() {
*
* @throws AllocationConfigurationException
*/
- public static Resource parseResourceConfigValue(String val)
+ public static ResourceConfiguration parseResourceConfigValue(String val)
throws AllocationConfigurationException {
+ ResourceConfiguration resourceConfiguration;
try {
val = StringUtils.toLowerCase(val);
- int memory = findResource(val, "mb");
- int vcores = findResource(val, "vcores");
- return BuilderUtils.newResource(memory, vcores);
+ if (val.contains("%")) {
+ resourceConfiguration = new ResourceConfiguration(
+ getResourcePercentage(val));
+ } else {
+ int memory = findResource(val, "mb");
+ int vcores = findResource(val, "vcores");
+ resourceConfiguration = new ResourceConfiguration(
+ BuilderUtils.newResource(memory, vcores));
+ }
} catch (AllocationConfigurationException ex) {
throw ex;
} catch (Exception ex) {
throw new AllocationConfigurationException(
"Error reading resource config", ex);
}
+ return resourceConfiguration;
+ }
+
+ private static HashMap<ResourceType, Double> getResourcePercentage(
+ String val) throws AllocationConfigurationException {
+ HashMap<ResourceType, Double> resourcePercentage = new HashMap<>();
+ String[] strings = val.split(",");
+ if (strings.length == 1) {
+ double percentage = findPercentage(strings[0], "");
+ for (ResourceType resourceType: ResourceType.values()) {
+ resourcePercentage.put(resourceType, percentage/100);
+ }
+ } else {
+ double memPercentage = findPercentage(val, "memory");
+ double vcorePercentage = findPercentage(val, "vcore");
--- End diff --
vcore is a unit of cpu. If we are using memory instead of mb, we should use
cpu instead of vcore.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -----------------------------------------------------------------------------------------
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
> Reporter: Ashwin Shankar
> Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
