[
https://issues.apache.org/jira/browse/MAPREDUCE-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-4963:
------------------------------------------
Resolution: Fixed
Fix Version/s: 1.2.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Robert. Committed to branch-1.
> StatisticsCollector improperly keeps track of "Last Day" and "Last Hour"
> statistics for new TaskTrackers
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4963
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4963
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
> Affects Versions: 1.1.1
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Fix For: 1.2.0
>
> Attachments: MAPREDUCE-4963.patch
>
>
> The StatisticsCollector keeps track of updates to the "Total Tasks Last Day",
> "Succeed Tasks Last Day", "Total Tasks Last Hour", and "Succeeded Tasks Last
> Hour" per Task Tracker which is displayed on the JobTracker web UI. It uses
> buckets to manage when to shift task counts from "Last Hour" to "Last Day"
> and out of "Last Day". After the JT has been running for a while, the
> connected TTs will have the max number of buckets and will keep shifting them
> at each update. If a new TT connects (or an old on rejoins), it won't have
> the max number of buckets, but the code that drops the buckets uses the same
> counter for all sets of buckets. This means that new TTs will prematurely
> drop their buckets and the stats will be incorrect.
> example:
> # Max buckets is 5
> # TaskTracker A has these values in its buckets [4, 2, 0, 3, 10] (i.e. 19)
> # A new TaskTracker, B, connects; it has nothing in its buckets: [ ] (i.e. 0)
> # TaskTracker B runs 3 tasks and TaskTracker A runs 5
> # An update occurs
> # TaskTracker A has [2, 0, 3, 10, 5] (i.e. 20)
> # TaskTracker B should have [3] but it will drop that bucket after adding it
> during the update and instead have [ ] again (i.e. 0)
> # TaskTracker B will keep doing that forever and always show 0 in the web UI
> We can fix this by not using the same counter for all sets of buckets
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
