[
https://issues.apache.org/jira/browse/MAPREDUCE-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15521338#comment-15521338
]
Haibo Chen commented on MAPREDUCE-6718:
---------------------------------------
Thanks for your reviews, [~rkanter].
bq. Something seems to be wrong with the math. When I had 0 jobs...
Sorry for the confusing phase. I guess a better way to put it is "x% of cache
is occupied." I believe the math is fine there. In the first case where you had
0 jobs, no jobs were loaded into the cache, so when the initialization was
completed, 0% of the cache was occupied. In the other case, the cache was never
more than 5% occupied, and the 20 jobs all-together just occupied 0.125% of the
total cache. The reason why you did not see the log message within the for-loop
is that the jobs (0, or 20) were too small.
bq. I also saw that the timing is not as useful. The idea is to print this out
while the JHS is loading files and appears to be stuck
The 10s gap you saw was taken before JHS started to initialize the job cache.
Specially, it was in the process of JHS trying to create directories in HDFS
(see createHistoryDirs(), there is a Thread.sleep()). I will add a if statement
in the createHistoryDirs method to avoid the unnecessary 10s sleep when the
operation is successful.
> add progress log to JHS during startup
> --------------------------------------
>
> Key: MAPREDUCE-6718
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6718
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Priority: Minor
> Labels: supportability
> Attachments: mapreduce6718.001.patch, mapreduce6718.002.patch
>
>
> lWhen the JHS starts up, it initializes the internal caches and storage via
> the HistoryFileManager. If we have a large number of existing finished jobs
> then we could spent minutes in this startup phase without logging progress:
> 2016-03-14 10:56:01,444 INFO
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hadoopcdh.itnas01.ieee.org:8020]
> 2016-03-14 10:56:11,455 INFO
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing
> Jobs...
> 2016-03-14 12:01:36,926 INFO
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage
> Init
> This makes it really difficult to assess if things are working correctly (it
> looks hung). We can add logs to notify users of progress.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
