[
https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207175#comment-16207175
]
Sunil G commented on YARN-7159:
-------------------------------
Thanks [[email protected]] for the effort.
I could see the improvement in removing {{UnitsConversionUtil.convert}} every
where for all internal computations. However by doing in
{{ResourceInformation.copy}} wont make all resource information in sync for
units.
To me, these are some entry points in RM where different units could come in
and we have to do the translation there.
# Over proto, clients and other yarn entities could send {{ResourceProto}}.
This will make RM to de-serialize and create the Resource object back with help
of ResourcePBImpl. Hence according to me, you also need to validate in
{{ResourceInformation newInstance}} where the incoming units is same as what
its there in ResourceUtils. Something like below may be ?
{code}
public static ResourceInformation newInstance(String name, String units,
long value, ResourceTypes type, long minimumAllocation,
long maximumAllocation) {
ResourceInformation ret = new ResourceInformation();
ret.setName(name);
ret.setResourceType(type);
if(!units.equals(ResourceUtils.getUnitsOfResourceXXX(name))) {
// call unit conversion to set Units and value
}
ret.setUnits(units);
ret.setValue(value);
ret.setMinimumAllocation(minimumAllocation);
ret.setMaximumAllocation(maximumAllocation);
return ret;
}
{code}
# I think we might need to check ResourceInformation.setUnits/setValue() as
well. But this will come with a performance hit. So lets check some other
alternatives as well here.
cc/ [~leftnoteasy]
> Normalize unit of resource objects in RM and avoid to do unit conversion in
> critical path
> -----------------------------------------------------------------------------------------
>
> Key: YARN-7159
> URL: https://issues.apache.org/jira/browse/YARN-7159
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Priority: Critical
> Attachments: YARN-7159.001.patch, YARN-7159.002.patch
>
>
> Currently resource conversion could happen in critical code path when
> different unit is specified by client. This could impact performance and
> throughput of RM a lot. We should do unit normalization when resource passed
> to RM and avoid expensive unit conversion every time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
