Wangda Tan commented on YARN-6789:

After a offline discussion with [~sunilg], I think we discovered more issues 
for unit in Resource object. 

The "unit" creates several issues:
- Behavior of the branch is: if unit of a given resource information is not 
set, default unit which configured in resource-type.cfg will be used to 
initialize containers ({{Resource.newInstance}}); and unit will be untouched 
for PB record initialization. ({{ResourcePBImpl(ResourceProto proto)}})
- However, if we have AM runs with old code (which doesn't have YARN-3926 
logics), it will send resource PB record to RM without unit on the wire. So 
YARN RM thinks the coming memory value with empty UNIT (which is B). This is an 
incompatible behavior. 
- Secondly, as I commented above, "unit" inside ResourceTypeInfo is very 
confusing: a. it is not minimum unit. b. it is not default unit, since it won't 
affect "default unit" inside AM, it is just default unit inside RM, which AM 
should not be interested. c. it is not a "suggested/preferred unit", because it 
doesn't make sense as well.
- In addition, It creates performance issue as well since all Resource 
operations need convert to the same unit.

My personal preference is completely removing unit from ResourceInformation. 
And unit of ResourceType means unit of given resource type. For example, 
resource.types.memory.unit = MB. It will be majorly used for UI displaying. 
Units of known resource types including vcores/memory will be hard coded in 
code and cannot changed by setting configuration file, this is majorly for 
backward-compatibility. We can provide unit converter as client library for 
AM/Client to use, Resource-related classes should not directly use it.


> new api to get all supported resources from RM
> ----------------------------------------------
>                 Key: YARN-6789
>                 URL: https://issues.apache.org/jira/browse/YARN-6789
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-6789-YARN-3926.001.patch
> It will be better to provide an api to get all supported resource types from 
> RM.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to