[
https://issues.apache.org/jira/browse/YARN-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952107#comment-15952107
]
Roni Burd commented on YARN-6245:
---------------------------------
We have run SLS for a couple of weeks, and we observed that when we have 4500 ,
the overhead of Resource accounts for a >60% of latencies.
Thought profiling, we found that this is because of 2 issues, some identified
here:
1) Resource is backed by protobuf, which is super expensive
2) Resource is treated 'most' of the time as immutable when doing calculations.
(for example, use of ResourceCalculator.add() instead of addTo(). T
Point #2 is especially hard. We observed that in our scenario, where we have
20K scheduler keys and 4500 machines, some of the methods like getHeaderoom or
computeUserLimits can get called almost 16million times, and if you look at the
method like nromalize() or normlizeAndRoundUp() etc, they each instantiate a
new Resource object. If you take that into account, you end up with <init> of
Resource, NetworkResource, etc getting called >40,000,000 times. This rate is
too high and it creates unnecessary OldGen GCs.
I would like to propose in addition to FinalResource a LightResource, that is
not only *not* backed by protobuf, but that it is also settable. In addition,
we would need to add some methods (like addTo/add pair) that instead of
creating a new object, simply set the properties of the old one to use in
several places in the capacity scheduler where math is being performed only to
be compared in an IF statement later.
Let me know what you think...
> Add FinalResource object to reduce overhead of Resource class instancing
> ------------------------------------------------------------------------
>
> Key: YARN-6245
> URL: https://issues.apache.org/jira/browse/YARN-6245
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Attachments: observable-resource.patch,
> YARN-6245.preliminary-staled.1.patch
>
>
> There're lots of Resource object creation in YARN Scheduler, since Resource
> object is backed by protobuf, creation of such objects is expensive and
> becomes bottleneck.
> To address the problem, we can introduce a FinalResource (Is it better to
> call it ImmutableResource?) object, which is not backed by PBImpl. We can use
> this object in frequent invoke paths in the scheduler.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
