[
https://issues.apache.org/jira/browse/MAPREDUCE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927090#action_12927090
]
Luke Lu commented on MAPREDUCE-2125:
------------------------------------
There are two major issues with the proposal/patch.
# The necessity of having these total aggregate counts in real time. Rumen or
other MR log processing tools can get these aggregates for performance analysis
without impacting JT performance.
# Have you tried to benchmark the patch at scale? Calling job.getCounters in
completeJob would bring down a busy JT on a large cluster to its knee. Think
about calling getCounters (which is essentially a O(n) operation) a few hundred
times per second!
If you really want these counters in real time, you should implement it in TT
where it can send the metrics to distributed metrics aggregators with UDP etc.
and can be easily disabled/enabled via the metrics system.
> Put map-reduce framework counters to JobTrackerMetricsInst
> ----------------------------------------------------------
>
> Key: MAPREDUCE-2125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2125
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-2125.txt
>
>
> We have lots of useful information in the framework counters including
> #spills, filesystem read and write.
> It will be nice to put them all in the jobtracker metrics to get a global
> view of all these numbers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
