I just submitted a Jira for this, with a patch against trunk.
Hadoop-3422 <https://issues.apache.org/jira/browse/HADOOP-3422>
If you pick up the patch, all of the metrics now have a leader in the
name indicating which context/job is doing the reporting, and all of the
job counters have their own rrd file and graph.
Jason Venner wrote:
I first verified that when I was using the file context, I saw the
counters in the report.
Then I switched context's to ganglia.
I also instrumented the low level code. I am hoping someone
understands this off the top of their head as I don't want to dig into
the metrics code to understand the difference ;)
Here is some syslog data. Note the missing name for the metric in the
JobInProgress case.
The JobTracker uses:
metricsRecord.incrMetric("maps_launched", numMapTasksLaunched);
metricsRecord.incrMetric("maps_completed", numMapTasksCompleted);
metricsRecord.incrMetric("reduces_launched",
numReduceTasksLaunched);
metricsRecord.incrMetric("reduces_completed",
numReduceTasksCompleted);
metricsRecord.incrMetric("jobs_submitted", numJobsSubmitted);
metricsRecord.incrMetric("jobs_completed", numJobsCompleted);
While JobInProgress uses:
for (String counter : group.getCounterNames()) {
long value = group.getCounter(counter);
jobMetrics.setTag("counter", group.getDisplayName(counter));
jobMetrics.setMetric("value", (float) value);
jobMetrics.update();
These are the job counters (JobInProgress)
2008-05-19 17:49:39,439 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
value, Type: float, Value: 36.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,439 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
value, Type: float, Value: 20.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,440 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
value, Type: float, Value: 0.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,447 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
value, Type: float, Value: 432.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,448 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
value, Type: float, Value: 0.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,448 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
value, Type: float, Value: 18.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,448 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
value, Type: float, Value: 576.0 to localhost/127.0.0.1:8649
These are the JobTracker counters
2008-05-19 17:49:39,449 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
jobs_completed, Type: int32, Value: 0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,449 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
jobs_submitted, Type: int32, Value: 1 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,449 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
maps_completed, Type: int32, Value: 18 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,449 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
maps_launched, Type: int32, Value: 20 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,450 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
reduces_completed, Type: int32, Value: 0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,450 INFO
org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name:
reduces_launched, Type: int32, Value: 0 to localhost/127.0.0.1:8649
--
Jason Venner
Attributor - Program the Web <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers and coding wizards, contact if
interested