I first verified that when I was using the file context, I saw the
counters in the report.
Then I switched context's to ganglia.
I also instrumented the low level code. I am hoping someone understands
this off the top of their head as I don't want to dig into the metrics
code to understand the difference ;)
Here is some syslog data. Note the missing name for the metric in the
JobInProgress case.
The JobTracker uses:
metricsRecord.incrMetric("maps_launched", numMapTasksLaunched);
metricsRecord.incrMetric("maps_completed", numMapTasksCompleted);
metricsRecord.incrMetric("reduces_launched",
numReduceTasksLaunched);
metricsRecord.incrMetric("reduces_completed",
numReduceTasksCompleted);
metricsRecord.incrMetric("jobs_submitted", numJobsSubmitted);
metricsRecord.incrMetric("jobs_completed", numJobsCompleted);
While JobInProgress uses:
for (String counter : group.getCounterNames()) {
long value = group.getCounter(counter);
jobMetrics.setTag("counter", group.getDisplayName(counter));
jobMetrics.setMetric("value", (float) value);
jobMetrics.update();
These are the job counters (JobInProgress)
2008-05-19 17:49:39,439 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: value, Type: float, Value: 36.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,439 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: value, Type: float, Value: 20.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,440 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: value, Type: float, Value: 0.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,447 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: value, Type: float, Value: 432.0 to
localhost/127.0.0.1:8649
2008-05-19 17:49:39,448 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: value, Type: float, Value: 0.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,448 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: value, Type: float, Value: 18.0 to localhost/127.0.0.1:8649
2008-05-19 17:49:39,448 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: value, Type: float, Value: 576.0 to
localhost/127.0.0.1:8649
These are the JobTracker counters
2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: jobs_completed, Type: int32, Value: 0 to
localhost/127.0.0.1:8649
2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: jobs_submitted, Type: int32, Value: 1 to
localhost/127.0.0.1:8649
2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: maps_completed, Type: int32, Value: 18 to
localhost/127.0.0.1:8649
2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: maps_launched, Type: int32, Value: 20 to
localhost/127.0.0.1:8649
2008-05-19 17:49:39,450 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: reduces_completed, Type: int32, Value: 0 to
localhost/127.0.0.1:8649
2008-05-19 17:49:39,450 INFO org.apache.hadoop.metrics.ganglia.GangliaContext:
Sending metric Name: reduces_launched, Type: int32, Value: 0 to
localhost/127.0.0.1:8649
--
Jason Venner
Attributor - Program the Web <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers and coding wizards, contact if
interested