[
https://issues.apache.org/jira/browse/YARN-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309840#comment-17309840
]
Qi Zhu edited comment on YARN-10503 at 3/27/21, 2:10 AM:
---------------------------------------------------------
Thanks [~ebadger] for good suggestion, i have updated in latest patch.
About the snippet of code , i will try to clarify:
*Firstly we will check if the resource name valid, the valid range has two :*
1. resourceTypes
Now resourceTypes is a first class resource enum set, added in following code:
{code:java}
for (AbsoluteResourceType type : AbsoluteResourceType.values()) {
resourceTypes.add(type.toString().toLowerCase());
}
/**
* Different resource types supported.
*/
public enum AbsoluteResourceType {
MEMORY, VCORES;
}
{code}
Only MEMORY and VCORES are first class resource.
2. ResourceUtils.getResourceTypes()
It will defined by user in resource-types.xml.
*We check the resourceName is valid in two ranges above.*
{code:java}
// If key is not a valid type, skip it.
if (!resourceTypes.contains(resourceName)
&& !ResourceUtils.getResourceTypes().containsKey(resourceName)) {
LOG.error(resourceName + " not supported.");
return;
}
{code}
*Secondly we check if it is in range defined by user in resource-types.xml.*
{code:java}
// Custom resource type defined by user.
// Such as GPU FPGA etc.
if (!resourceTypes.contains(resourceName)) {
resource.setResourceInformation(resourceName, ResourceInformation
.newInstance(resourceName, units, resourceValue));
return;
}
{code}
*Lastly we check if it is in first class enum set:*
{code:java}
// map it based on key.
AbsoluteResourceType resType = AbsoluteResourceType
.valueOf(StringUtils.toUpperCase(resourceName));
switch (resType) {
case MEMORY :
resource.setMemorySize(resourceValue);
break;
case VCORES :
resource.setVirtualCores(resourceValue.intValue());
break;
default :
resource.setResourceInformation(resourceName, ResourceInformation
.newInstance(resourceName, units, resourceValue));
break;
}
{code}
And why we add default here, because we reserved the other first class resource
may added in future.
And why the design for first class enum, not just used the
{code:java}
// Known resource types
public static final String MEMORY_URI = "memory-mb";
public static final String VCORES_URI = "vcores";
{code}
I think, we used to make memory simple to support memory also, not just
memory-mb.
Thanks.
was (Author: zhuqi):
Thanks [~ebadger] for good suggestion, i have updated in latest patch.
About the code, i will try to clarify:
*Firstly we will check if the resource name valid, the valid range has two :*
1. resourceTypes
Now resourceTypes is a first class resource enum set, added in following code:
{code:java}
for (AbsoluteResourceType type : AbsoluteResourceType.values()) {
resourceTypes.add(type.toString().toLowerCase());
}
/**
* Different resource types supported.
*/
public enum AbsoluteResourceType {
MEMORY, VCORES;
}
{code}
Only MEMORY and VCORES are first class resource.
2. ResourceUtils.getResourceTypes()
It will defined by user in resource-types.xml.
*We check the resourceName is valid in two ranges above.*
{code:java}
// If key is not a valid type, skip it.
if (!resourceTypes.contains(resourceName)
&& !ResourceUtils.getResourceTypes().containsKey(resourceName)) {
LOG.error(resourceName + " not supported.");
return;
}
{code}
*Secondly we check if it is in range defined by user in resource-types.xml.*
{code:java}
// Custom resource type defined by user.
// Such as GPU FPGA etc.
if (!resourceTypes.contains(resourceName)) {
resource.setResourceInformation(resourceName, ResourceInformation
.newInstance(resourceName, units, resourceValue));
return;
}
{code}
*Lastly we check if it is in first class enum set:*
{code:java}
// map it based on key.
AbsoluteResourceType resType = AbsoluteResourceType
.valueOf(StringUtils.toUpperCase(resourceName));
switch (resType) {
case MEMORY :
resource.setMemorySize(resourceValue);
break;
case VCORES :
resource.setVirtualCores(resourceValue.intValue());
break;
default :
resource.setResourceInformation(resourceName, ResourceInformation
.newInstance(resourceName, units, resourceValue));
break;
}
{code}
And why we add default here, because we reserved the other first class resource
may added in future.
And why the design for first class enum, not just used the
{code:java}
// Known resource types
public static final String MEMORY_URI = "memory-mb";
public static final String VCORES_URI = "vcores";
{code}
I think, we used to make memory simple to support memory also, not just
memory-mb.
Thanks.
> Support queue capacity in terms of absolute resources with custom
> resourceType.
> -------------------------------------------------------------------------------
>
> Key: YARN-10503
> URL: https://issues.apache.org/jira/browse/YARN-10503
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Qi Zhu
> Assignee: Qi Zhu
> Priority: Critical
> Attachments: YARN-10503.001.patch, YARN-10503.002.patch,
> YARN-10503.003.patch, YARN-10503.004.patch, YARN-10503.005.patch,
> YARN-10503.006.patch, YARN-10503.007.patch, YARN-10503.008.patch
>
>
> Now the absolute resources are memory and cores.
> {code:java}
> /**
> * Different resource types supported.
> */
> public enum AbsoluteResourceType {
> MEMORY, VCORES;
> }{code}
> But in our GPU production clusters, we need to support more resourceTypes.
> It's very import for cluster scaling when with different resourceType
> absolute demands.
>
> This Jira will handle GPU first.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
