[
https://issues.apache.org/jira/browse/MAPREDUCE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danica Zhang updated MAPREDUCE-5453:
------------------------------------
Description:
It seems that FILE_BYTES_READ/FILE_BYTES_WRITTEN can NOT be updated in
streaming application by:
reporter:counter:<group>,<counter>,<amount>
The syntax I tried is:
System.err.println("reporter:counter:"+ "FileSystemCounters" + ","
+ "FILE_BYTES_READ" + ","
+ number);
Since File System counters are added later into the framework by ticket
https://issues.apache.org/jira/browse/HADOOP-3001
and that's why it is NOT properly handled by streaming.jar.
The implementation of File System counter is a bit different, and it keeps
track of the real information in FileSystem.Statistics object. When the task
finished, the framework would read the info from 'FileSystem.Statistics' and
set the File System counters accordingly.
Therefore, the regular way to update counters -- ' findCounter() and
Counter.incrment() ' would NOT work for the File System counters as they would
be overwritten by the info in 'FileSystem.Statistics ' eventually.
However, it seems that the 'reporter:counter:' scheme in streaming.jar is
implemented by the regular way, and maybe that's why it is not working.
was:
It seems that FILE_BYTES_READ/FILE_BYTES_WRITTEN can NOT be updated in
streaming application by:
reporter:counter:<group>,<counter>,<amount>
The syntax I tried is:
System.err.println("reporter:counter:"+ "FileSystemCounters" + ","
+ "FILE_BYTES_READ" + ","
+ number);
Since File System counters are added later into the framework by ticket
https://issues.apache.org/jira/browse/HADOOP-3001
and that's why it is NOT properly handled by streaming.jar.
The implementation of File System counter is a bit different, and it keeps
track of the real information in FileSystem.Statistics object. When the task
finished, it framework would read the info from 'FileSystem.Statistics' and set
the File System counters accordingly.
Therefore, the regular way to update counters -- ' findCounter() and
Counter.incrment() ' would NOT work for the File System counters as they would
be overwritten by the info in 'FileSystem.Statistics ' eventually.
However, it seems that the 'reporter:counter:' scheme in streaming.jar is
implemented by the regular way, and maybe that's why it is not working.
> File System Counters can NOT be updated in streaming application
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-5453
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5453
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Danica Zhang
>
> It seems that FILE_BYTES_READ/FILE_BYTES_WRITTEN can NOT be updated in
> streaming application by:
> reporter:counter:<group>,<counter>,<amount>
> The syntax I tried is:
> System.err.println("reporter:counter:"+ "FileSystemCounters" + ","
> + "FILE_BYTES_READ" + ","
> + number);
> Since File System counters are added later into the framework by ticket
> https://issues.apache.org/jira/browse/HADOOP-3001
>
> and that's why it is NOT properly handled by streaming.jar.
> The implementation of File System counter is a bit different, and it keeps
> track of the real information in FileSystem.Statistics object. When the task
> finished, the framework would read the info from 'FileSystem.Statistics' and
> set the File System counters accordingly.
> Therefore, the regular way to update counters -- ' findCounter() and
> Counter.incrment() ' would NOT work for the File System counters as they
> would be overwritten by the info in 'FileSystem.Statistics ' eventually.
> However, it seems that the 'reporter:counter:' scheme in streaming.jar is
> implemented by the regular way, and maybe that's why it is not working.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira