[
https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16057718#comment-16057718
]
Daryn Sharp commented on YARN-6679:
-----------------------------------
[~jlowe] or [~nroberts] may be able to comment on the allocation throughput. I
just reduced overhead found by a profiler.
SLS may not be exercising the RM in the same manner as in a real-world setting.
If you look at {{ResourcePBImpl}} it has to:
# instantiate a builder – wasted object
# builder and its parent class have unneeded instance variables – wasted memory
# call setters for memory and vcores, each updates a bit field, assigns
instance variable, marks parent builder dirty – unnecessary computational
overhead
By comparison, a simple object with 2 longs is clearly a win. Even if you
aren't stressing the scheduler to its maximum, you should see fewer gc/min due
to slower heap growth. I don't have the profile available but the cost of
excessive Resource instantiations is still a non-trivial percent of the loop.
> Reduce Resource instance overhead via non-PBImpl
> ------------------------------------------------
>
> Key: YARN-6679
> URL: https://issues.apache.org/jira/browse/YARN-6679
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 2.8.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch,
> YARN-6679.3.branch-2.patch, YARN-6679.3.trunk.patch,
> YARN-6679.branch-2.patch, YARN-6679.trunk.patch
>
>
> Creating and using transient PB-based Resource instances during scheduling is
> very expensive. The overhead can be transparently reduced by internally
> using lightweight non-PB based instances.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
