[
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669336#comment-13669336
]
Jason Lowe commented on MAPREDUCE-5268:
---------------------------------------
bq. Is the ordering a requirement? If it is not, we could may be use something
like GuavaCache?
Ordering is important for the code that evicts entries from the cache. The
only other access that isn't a simple put/get is in
CachedHistoryStorage.getAllPartialJobs, but that method explicitly builds a
SortedMap separately and returns that.
Therefore I think using a Guava cache is possible, although it wouldn't be as
straightforward as it might initially seem. I think we'd have to use a
weighted cache with the weight being an increasing sequence number so it
removes entries based on cache size but in insertion order rather than LRU.
Also we'd need a removal listener to replicate the special-case cache removal
logic when a move is still pending or has failed, and have a way to follow up
on history file infos that need to be flushed from the cache but the move is
still pending. It's all possible, just not as simple as dropping in the Guava
cache and declaring victory.
I'm OK with the current approach with one suggestion: I'd prefer the
size/collection pair implementation detail abstracted into a local utility
class. That way as the code is maintained someone can't easily add or remove
elements from the collection and forget to update the separate size variable.
> Improve history server startup performance
> ------------------------------------------
>
> Key: MAPREDUCE-5268
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
> Affects Versions: 0.23.7, 2.0.4-alpha
> Reporter: Jason Lowe
> Assignee: Karthik Kambatla
> Attachments: mr-5268.patch, mr-5268-prelim.patch
>
>
> The history server can easily take many minutes to startup when there are a
> significant number of jobs to scan in the done directory. However the
> scanning of files is not the bottleneck, rather it's the heavy use of
> ConcurrentSkipListMap.size in HistoryFileManager.
> ConcurrentSkipListMap.size is a very expensive operation, especially on maps
> with many entries, as it has to scan every entry to compute the size. We
> should avoid calling this method or at least minimize its use.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
