[
https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207301#comment-13207301
]
Todd Lipcon commented on MAPREDUCE-2855:
----------------------------------------
This is taking the majority of time when loading job history for large jobs. I
see the following stack:
{code}
Stack:
java.util.ResourceBundle.getClassContext(Native Method)
java.util.ResourceBundle.getLoader(ResourceBundle.java:400)
java.util.ResourceBundle.getBundle(ResourceBundle.java:705)
org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:36)
org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:54)
org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterName(ResourceBundles.java:87)
org.apache.hadoop.mapreduce.counters.FileSystemCounterGroup$FSCounter.localizeCounterName(FileSystemCounterGroup.java:85)
org.apache.hadoop.mapreduce.counters.FileSystemCounterGroup$FSCounter.getDisplayName(FileSystemCounterGroup.java:81)
org.apache.hadoop.mapreduce.TypeConverter.toYarn(TypeConverter.java:275)
org.apache.hadoop.mapreduce.v2.hs.CompletedTask.<init>(CompletedTask.java:89)
org.apache.hadoop.mapreduce.v2.hs.CompletedJob.loadFullHistoryData(CompletedJob.java:270)
org.apache.hadoop.mapreduce.v2.hs.CompletedJob.<init>(CompletedJob.java:90)
org.apache.hadoop.mapreduce.v2.hs.JobHistory.loadJob(JobHistory.java:657)
org.apache.hadoop.mapreduce.v2.hs.JobHistory.findJob(JobHistory.java:737)
org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:903)
org.apache.hadoop.mapreduce.v2.app.webapp.AppController.requireJob(AppController.java:313)
{code}
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
>
> Loading a job status page in trunk takes a lot of time, and it seems like
> most of the time is spent resolving counter names. Looking through the JDK
> source, ResourceBundle.getBundle(String) ends up calling getClassContext()
> which is not very efficient. I think if we pass our own classloader manually
> it will be faster. In Counters.incrAllCounters, we may also be able to avoid
> setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
