[
https://issues.apache.org/jira/browse/MAPREDUCE-2124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919940#action_12919940
]
Scott Chen commented on MAPREDUCE-2124:
---------------------------------------
Hey Chris,
We can put this in job counter (the one in JobInProgress) instead of task
counter.
So these counters will be recorded per job. The memory cost should be very
minor.
Also the cost for collecting them is very low. We can do this in
JobInProgress.meterTaskAttempt() which will be called only once after an
attempt is finished.
These times are recorded in job history but it will be nice to have them on the
web UI if there is no much cost.
What do you think?
Scott
> Add job counters for measuring time spent in three different phases in
> reducers
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2124
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2124
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Priority: Minor
> Fix For: 0.22.0
>
>
> We currently have SLOTS_MILLIS_REDUCES which measures the total slot time of
> reducer.
> It will be useful if we have
> {code}
> SLOTS_MILLIS_REDUCES_COPY
> SLOTS_MILLIS_REDUCES_SORT
> SLOTS_MILLIS_REDUCES_REDUCE
> {code}
> which measures three different phases of a reducer.
> This will help us identify the bottleneck of the reducers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
