[
https://issues.apache.org/jira/browse/HADOOP-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526803#comment-16526803
]
Todd Lipcon commented on HADOOP-15554:
--------------------------------------
I think the discrepancy may be related to the total number of configuration
elements in your conf files, as well as how many times confs are loaded. Once
the loadResources call has hit a certain number of iterations, it will get
compiled by the C2 compiler, at which point you'll see that slow compilation
prior to the patch. As you can see in my measurements, quick test programs and
"hadoop fs" command lines don't trigger this, but running a simple job using
LocalJobRunner did. We also saw this causing perf issues in a "sleep" MR job
submitted to YARN, but only on some clusters (where the number of configuration
elements was large enough, perhaps).
We were planning on backporting this downstream into CDH so we can probably
provide a backport for 3.0. I don't think we would backport to branch-2, though.
> Improve JIT performance for Configuration parsing
> -------------------------------------------------
>
> Key: HADOOP-15554
> URL: https://issues.apache.org/jira/browse/HADOOP-15554
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, performance
> Affects Versions: 3.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hadoop-15554.patch, hadoop-15554.patch
>
>
> In investigating a performance regression for small tasks between Hadoop 2
> and Hadoop 3, we found that the amount of time spent in JIT was significantly
> higher. Using jitwatch we were able to determine that, due to a combination
> of switching from DOM to SAX style parsing and just having more configuration
> key/value pairs, Configuration.loadResource is now getting compiled with the
> C2 compiler and taking quite some time. Breaking that very large function up
> into several smaller ones and eliminating some redundant bits of code
> improves the JIT performance measurably.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
