[jira] [Commented] (YARN-10503) Support queue capacity in terms of absolute resources with gpu resourceType.

[Peter Bacsko (Jira)]

    [ 
https://issues.apache.org/jira/browse/YARN-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306157#comment-17306157
 ] 

Peter Bacsko commented on YARN-10503:
-------------------------------------

The question is this part:

{noformat}
public enum AbsoluteResourceType {
    MEMORY, VCORES, GPUS, FPGAS
}
{noformat}

Do we want to treat GPUs and FPGAs like that? In other parts of the code, we 
have mem/vcore as primary resources, then an array of other resources.  For 
example, constructors from {{org.apache.hadoop.yarn.api.records.Resource}}:

{noformat}
  @Public
  @Stable
  public static Resource newInstance(long memory, int vCores,
      Map<String, Long> others) {
    if (others != null) {
      return new LightWeightResource(memory, vCores,
          ResourceUtils.createResourceTypesArray(others));
    } else {
      return newInstance(memory, vCores);
    }
  }

  @InterfaceAudience.Private
  @InterfaceStability.Unstable
  public static Resource newInstance(Resource resource) {
    Resource ret;
    int numberOfKnownResourceTypes = ResourceUtils
        .getNumberOfKnownResourceTypes();
    if (numberOfKnownResourceTypes > 2) {
      ret = new LightWeightResource(resource.getMemorySize(),
          resource.getVirtualCores(), resource.getResources());
    } else {
      ret = new LightWeightResource(resource.getMemorySize(),
          resource.getVirtualCores());
    }
    return ret;
  }
{noformat}

But with this modification, we sort of promote GPU and FPGA to the level of 
vcore and memory, at least from the perspective of the code and it also becomes 
inconsistent with the existing code.

This is just my opinion though. cc [~epayne] [~ebadger].

> Support queue capacity in terms of absolute resources with gpu resourceType.
> ----------------------------------------------------------------------------
>
>                 Key: YARN-10503
>                 URL: https://issues.apache.org/jira/browse/YARN-10503
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Qi Zhu
>            Assignee: Qi Zhu
>            Priority: Critical
>         Attachments: YARN-10503.001.patch, YARN-10503.002.patch, 
> YARN-10503.003.patch
>
>
> Now the absolute resources are memory and cores.
> {code:java}
> /**
>  * Different resource types supported.
>  */
> public enum AbsoluteResourceType {
>   MEMORY, VCORES;
> }{code}
> But in our GPU production clusters, we need to support more resourceTypes.
> It's very import for cluster scaling when with different resourceType 
> absolute demands.
>  
> This Jira will handle GPU first.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org