Danica Zhang created MAPREDUCE-5453:
---------------------------------------
Summary: File System Counters can NOT be updated in streaming
application
Key: MAPREDUCE-5453
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5453
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Danica Zhang
It seems that FILE_BYTES_READ/FILE_BYTES_WRITTEN can NOT be updated in
streaming application by:
reporter:counter:<group>,<counter>,<amount>
The syntax I tried is:
System.err.println("reporter:counter:"+ "FileSystemCounters" + ","
+ "FILE_BYTES_READ" + ","
+ number);
Since File System counters are added later into the framework by ticket
https://issues.apache.org/jira/browse/HADOOP-3001
and that's why it is NOT properly handled by streaming.jar.
The implementation of File System counter is a bit different, and it keeps
track of the real information in FileSystem.Statistics object. When the task
finished, it framework would read the info from 'FileSystem.Statistics' and set
the File System counters accordingly.
Therefore, the regular way to update counters -- ' findCounter() and
Counter.incrment() ' would NOT work for the File System counters as they would
be overwritten by the info in 'FileSystem.Statistics ' eventually.
However, it seems that the 'reporter:counter:' scheme in streaming.jar is
implemented by the regular way, and maybe that's why it is not working.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
