[
https://issues.apache.org/jira/browse/MAPREDUCE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927147#action_12927147
]
Luke Lu commented on MAPREDUCE-2125:
------------------------------------
bq. You are right. The getCounter() will happen in completeJob(). But it is
called once per job for the entire life-cycle of one job. So it has very minor
impact on JT performance.
What's the job completion rate on your cluster? If you just run a few big jobs
that takes a while, It probably doesn't matter. But if you have jobs that takes
seconds to minutes rather than hours (typically submitted via automated systems
and getCounters is not even called because they're intermediate steps) the job
completion rate can be very high, I've seen over 100 per second on our clusters.
I'd be more comfortable with the patch, if we have an easy way to disable the
expensive metrics. One way to do it without a lot of ifs is to create an
alternative instrumentation class that subclass from the default class.
> Put map-reduce framework counters to JobTrackerMetricsInst
> ----------------------------------------------------------
>
> Key: MAPREDUCE-2125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2125
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-2125.txt
>
>
> We have lots of useful information in the framework counters including
> #spills, filesystem read and write.
> It will be nice to put them all in the jobtracker metrics to get a global
> view of all these numbers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
