[
https://issues.apache.org/jira/browse/YARN-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306170#comment-17306170
]
Qi Zhu commented on YARN-10503:
-------------------------------
Thanks [~pbacsko] for review.
I add this to support more string types to support gpus and fpgas:
{code:java}
public enum AbsoluteResourceType {
MEMORY, VCORES, GPUS, FPGAS
}
{code}
Actually, we can also use the original case in the conf:
{code:java}
public static final String MEMORY_URI = "memory-mb";
public static final String VCORES_URI = "vcores";
public static final String GPU_URI = "yarn.io/gpu";
public static final String FPGA_URI = "yarn.io/fpga";
{code}
such as :
{code:java}
1. <memory=10,vcores=10,gpus=10,fpgas=10>
2. <memory-mb=10,vcores=10,yarn.io/gpu=10,yarn.io/fpga=10>{code}
It will also take effect both two situation in latest patch, the corresponding
logic:
{code:java}
2. <memory-mb=10,vcores=10,yarn.io/gpu=10,yarn.io/fpga=10>
// Custom resource type defined by user.
if (!resourceTypes.contains(splits[0])) {
resource.setResourceInformation(splits[0].trim(), ResourceInformation
.newInstance(splits[0].trim(), units, resourceValue));
return;
}
1. <memory=10,vcores=10,gpus=10,fpgas=10>
// map it based on key.
AbsoluteResourceType resType = AbsoluteResourceType
.valueOf(StringUtils.toUpperCase(splits[0].trim()));
switch (resType) {
case MEMORY :
resource.setMemorySize(resourceValue);
break;
case VCORES :
resource.setVirtualCores(resourceValue.intValue());
break;
case GPUS :
Integer gpuIndex = ResourceUtils.getResourceTypeIndex()
.get(ResourceInformation.GPU_URI);
if (gpuIndex != null) {
resource.setResourceValue(ResourceInformation.GPU_URI,
resourceValue.intValue());
} else {
LOG.error("GPU is not supported in conf.");
}
break;
case FPGAS :
Integer fpgaIndex = ResourceUtils.getResourceTypeIndex()
.get(ResourceInformation.FPGA_URI);
if (fpgaIndex != null) {
resource.setResourceValue(ResourceInformation.FPGA_URI,
resourceValue.intValue());
} else {
LOG.error("FPGA is not supported in conf.");
}
break;
default :
resource.setResourceInformation(splits[0].trim(), ResourceInformation
.newInstance(splits[0].trim(), units, resourceValue));
break;
}
{code}
> Support queue capacity in terms of absolute resources with gpu resourceType.
> ----------------------------------------------------------------------------
>
> Key: YARN-10503
> URL: https://issues.apache.org/jira/browse/YARN-10503
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Qi Zhu
> Assignee: Qi Zhu
> Priority: Critical
> Attachments: YARN-10503.001.patch, YARN-10503.002.patch,
> YARN-10503.003.patch
>
>
> Now the absolute resources are memory and cores.
> {code:java}
> /**
> * Different resource types supported.
> */
> public enum AbsoluteResourceType {
> MEMORY, VCORES;
> }{code}
> But in our GPU production clusters, we need to support more resourceTypes.
> It's very import for cluster scaling when with different resourceType
> absolute demands.
>
> This Jira will handle GPU first.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
