[
https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218052#comment-16218052
]
Manikandan R commented on YARN-7159:
------------------------------------
[~leftnoteasy] Thanks for your review.
{quote}I agree with what Sunil G suggested. Instead of doing this on RM's
deserialization part, we should do this in ResourceInformation#newInstance. The
reason is: Resources class by definition, can be used by map reduce as well. It
will not be enough to only change RM's deserialization path and remove all unit
conversions in Resources.{quote}
Yes, I agree. [~sunilg] also had explained the same in offline discussion.
{quote}I also agree that changing setUnits to check unit might be a perf issue
since it is invoked by copy. Can we add an internal setUnitsWithoutChecking
method, which will be used by copy, and when setUnits is invoked, instead of
setting unit, we will change value. For example, assume a resource
information's value is 100, unit is MB (which is same as
ResourceUtils.getUnitsOfResourceXXX), if user calls setUnit(GB). Instead of
changing unit to GB, we will keep the unit to MB and change value to 100 *
1024. And we can avoid the expensive setUnit in all RM's code paths.{quote}
Based on my understaning, it has two parts. 1) Changes in
{{ResourceInformation#copy}} 2)Changes in {{ResourceInformation#setUnits}}. #2
changes would be something like
public void setUnits(String rUnits) {
String unit = ResourceUtils.getDefaultUnit(this.getName());
if(!rUnits.equals(unit)) {
this.setValue(
UnitsConversionUtil.convert(rUnits, unit, this.getValue()));
}
this.unit = rUnits;
}
Can I assume #1 changes are reasonable since we would be making changes in
{{ResourceInformation#newInstance}} to handle conversion if required? Also, how
are we going to ensure units validation coming from users if
{{ResourceInformation#setUnits}} would be modified based as given above? For
example, users may be passing "abc", xyz". or Is it ok to simply ignore those
example inputs given that we would be doing conversion anyways if there is
difference?
Please correct my understanding.
> Normalize unit of resource objects in RM and avoid to do unit conversion in
> critical path
> -----------------------------------------------------------------------------------------
>
> Key: YARN-7159
> URL: https://issues.apache.org/jira/browse/YARN-7159
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Priority: Critical
> Attachments: YARN-7159.001.patch, YARN-7159.002.patch
>
>
> Currently resource conversion could happen in critical code path when
> different unit is specified by client. This could impact performance and
> throughput of RM a lot. We should do unit normalization when resource passed
> to RM and avoid expensive unit conversion every time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
