[
https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217556#comment-16217556
]
Wangda Tan commented on YARN-7159:
----------------------------------
Thanks [[email protected]] for working on this.
I agree with what [~sunilg] suggested. Instead of doing this on RM's
deserialization part, we should do this in ResourceInformation#newInstance. The
reason is: {{Resources}} class by definition, can be used by map reduce as
well. It will not be enough to only change RM's deserialization path and remove
all unit conversions in Resources.
I also agree that changing {{setUnits}} to check unit might be a perf issue
since it is invoked by {{copy}}. Can we add an internal
{{setUnitsWithoutChecking}} method, which will be used by {{copy}}, and when
{{setUnits}} is invoked, instead of setting unit, we will change value. For
example, assume a resource information's value is 100, unit is MB (which is
same as {{ResourceUtils.getUnitsOfResourceXXX}}), if user calls setUnit(GB).
Instead of changing unit to GB, we will keep the unit to MB and change value to
100 * 1024. And we can avoid the expensive {{setUnit}} in all RM's code paths.
Thoughts?
> Normalize unit of resource objects in RM and avoid to do unit conversion in
> critical path
> -----------------------------------------------------------------------------------------
>
> Key: YARN-7159
> URL: https://issues.apache.org/jira/browse/YARN-7159
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Priority: Critical
> Attachments: YARN-7159.001.patch, YARN-7159.002.patch
>
>
> Currently resource conversion could happen in critical code path when
> different unit is specified by client. This could impact performance and
> throughput of RM a lot. We should do unit normalization when resource passed
> to RM and avoid expensive unit conversion every time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
