[
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670930#comment-13670930
]
Jason Lowe commented on MAPREDUCE-5268:
---------------------------------------
bq. Considered implementing a full-blown implementation of
ConcurrentSkipListMapWithSize extending ConcurrentSkipList and adding it to
hadoop-common, but decided against it as that seemed a little overboard.
Agreed. Actually I'm a bit surprised to see a templated interface for the
private class. The whole purpose of the class is to be a map of JobId to
HistoryFileInfo so templating seems a bit overkill. Not a big deal, though.
I was a bit puzzled to see the synchronized methods instead of the
AtomicInteger. What was the reasoning behind that change? It seems a bit odd
to see a ConcurrentSkipListMap behind synchronized accesses.
BTW, I built a local test of 50,000 jhist files with the history server
configured to cache 20,000. Without this change the history server takes over
60 seconds to process the existing jobs, and after the patch it processes them
in around 6 seconds.
> Improve history server startup performance
> ------------------------------------------
>
> Key: MAPREDUCE-5268
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
> Affects Versions: 0.23.7, 2.0.4-alpha
> Reporter: Jason Lowe
> Assignee: Karthik Kambatla
> Attachments: mr-5268.patch, mr-5268.patch, mr-5268-prelim.patch
>
>
> The history server can easily take many minutes to startup when there are a
> significant number of jobs to scan in the done directory. However the
> scanning of files is not the bottleneck, rather it's the heavy use of
> ConcurrentSkipListMap.size in HistoryFileManager.
> ConcurrentSkipListMap.size is a very expensive operation, especially on maps
> with many entries, as it has to scan every entry to compute the size. We
> should avoid calling this method or at least minimize its use.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
