HI,
We found crunch job has no counters outputted into the log. Tried to output
logs by following code, but still, those counters emitted by InputFormat
classes and those default map reduce counters were missing. How can I see those
counters a normal map reduce job will produce?
public static PipelineResult printCounters(Pipeline pipeline) {
PipelineResult result = pipeline.run();
List<StageResult> results = result.getStageResults();
for (StageResult partial : results) {
String stageId = partial.getStageName();
log.info(String.format("counters for %s :", stageId));
for (Map.Entry<String, Set<String>> entry :
partial.getCounterNames().entrySet()){
String groupName = entry.getKey();
log.info(String.format(" %s :", groupName));
for (String counterName : entry.getValue()) {
long value = partial.getCounterValue(groupName, counterName);
log.info(String.format(" %s : %d", counterName, value));
}
}
}
return result;
}
I’m expecting to see counters like this, and all counters emitted from input
format class instead of only those emitted by Crunch:
Map-Reduce Framework
Map input records= 1234567890
Map output records= 1234567
Map output bytes=1234598760
Thanks!