[
https://issues.apache.org/jira/browse/MAPREDUCE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920440#action_12920440
]
Kang Xiao commented on MAPREDUCE-2026:
--------------------------------------
+1 even JIP lock is removed for aggregating tips counters
> From our cluster, abount 26% of jobtracker's lock taken by the jsp access.
In fact, we decrease lock time by cache getCounters() result. During job is
running getCounters are only 1min fresh while it will be completely accurate
when job is finished. However this just decrease the invokation for aggregating
tips counters and JIP is still in lock when getCounters() is called if the
cache is out of 1min.
Maybe we can combine getCounters() cache and no-lock for aggregating tips
counters, than both CPU and lock consumptioncan be decreased. Is that OK?
> JobTracker.getJobCounters() should not hold JobTracker lock while calling
> JobInProgress.getCounters()
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2026
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Scott Chen
> Assignee: Joydeep Sen Sarma
> Fix For: 0.22.0
>
> Attachments: 2026.1.patch, MAPREDUCE-2026.txt
>
>
> JobTracker.getJobCounter() will lock JobTracker and call
> JobInProgress.getCounters().
> JobInProgress.getCounters() can be very expensive because it aggregates all
> the task counters.
> We found that from the JobTracker jstacks that this method is one of the
> bottleneck of the JobTracker performance.
> JobInProgress.getCounters() should be able to be called out side the
> JobTracker lock because it already has JobInProgress lock.
> For example, it is used by jobdetails.jsp without a JobTracker lock.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
