[
https://issues.apache.org/jira/browse/MAPREDUCE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923383#action_12923383
]
Ravi Gummadi commented on MAPREDUCE-2135:
-----------------------------------------
Ah! I see that the map output bytes and map output records shown are the
counters that represent the values before the combiner. Combine input records
and Combine output records are shown in the counters.
May be it would be better to have counter for combine output bytes also,
because that is the size of the intermediate file generated by map task ?
> FILE_BYTES_WRITTEN counter in map task seems incorrect
> ------------------------------------------------------
>
> Key: MAPREDUCE-2135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Reporter: Ravi Gummadi
>
> With MapReduce trunk,
> The FileSystem counter FILE_BYTES_WRITTEN is a lot less than "Map output
> bytes" counter even when map output compression is OFF. I think this
> FILE_BYTES_WRITTEN signifies the bytes written to local file system. So it
> should be more than map output bytes(in the counters shown below, 210 Vs
> 19200000). Right ?
> Here are some counters from map task of wordcount example:
> Counters for attempt_201010141448_0001_m_000000_0
> FileInputFormatCounters
> BYTES_READ 9,600,000
> FileSystemCounters
> FILE_BYTES_READ 92
> FILE_BYTES_WRITTEN 210
> HDFS_BYTES_READ 9,600,107
> Map-Reduce Framework
> Combine input records 2,400,000
> Combine output records 8
> CPU_MILLISECONDS 4,810
> Failed Shuffles 0
> GC time elapsed (ms) 73
> Map input records 600,000
> Map output bytes 19,200,000
> Map output records 2,400,000
> Merged Map outputs 0
> PHYSICAL_MEMORY_BYTES 131,518,464
> Spilled Records 16
> SPLIT_RAW_BYTES 107
> VIRTUAL_MEMORY_BYTES 581,021,696
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
