[
https://issues.apache.org/jira/browse/MAPREDUCE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927239#action_12927239
]
Scott Chen commented on MAPREDUCE-2125:
---------------------------------------
Hey Luke,
{quote}
What's the job completion rate on your cluster? If you just run a few big jobs
that takes a while, It probably doesn't matter. But if you have jobs that takes
seconds to minutes rather than hours (typically submitted via automated systems
and getCounters is not even called because they're intermediate steps) the job
completion rate can be very high, I've seen over 100 per second on our clusters.
{quote}
If the use case is for many small jobs, each getCounter() call will be cheap.
So in this case it will still be OK.
I think the key here is that this change only add one more look at all task
counter. So from the throughput point of view, it is not that large.
Our job completion rate is about 20 jobs/minute in average.
Comparing to the general use case in our cluster that HIVE pulls the
jobCounters periodically, this one is very light weight because it does
getCounters only once.
> Put map-reduce framework counters to JobTrackerMetricsInst
> ----------------------------------------------------------
>
> Key: MAPREDUCE-2125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2125
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-2125.txt
>
>
> We have lots of useful information in the framework counters including
> #spills, filesystem read and write.
> It will be nice to put them all in the jobtracker metrics to get a global
> view of all these numbers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
