[
https://issues.apache.org/jira/browse/MAPREDUCE-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969152#action_12969152
]
Joydeep Sen Sarma commented on MAPREDUCE-2212:
----------------------------------------------
Todd - do you know for sure that the benefit is due to the compression of the
final spill or because of the compression of the intermediate sort runs? i am
thinking that if the experiment is just to turn compression on/off and run some
benchmark - then it wouldn't be clear whether any win is from lower network
latencies (from map->reduce) or from faster mappers (if they were disk bound
without compression).
in general i have seen that the map-reduce stack consumes data at a very low
rate (it's cpu bound by the time it gets to 10-20 MBps). (Obviously this is a
very loose statement and depends a lot on what the mappers are doing etc.). so
even with 6 disks (say a total of 300MBps streaming read/write bandwidth) and
8 cores (say about 200 MBps processing bandwidth) - it would seem that we would
be cpu bound before we would be disk throughput bound. would be nice to get
more accurate numbers along these lines.
> MapTask and ReduceTask should only compress/decompress the final map output
> file
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2212
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2212
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.23.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.23.0
>
>
> Currently if we set mapred.map.output.compression.codec
> 1. MapTask will compress every spill, decompress every spill, merge and
> compress the final map output file
> 2. ReduceTask will decompress, merge and compress every map output file. And
> repeat the compression/decompression every pass.
> This causes all the data being compressed/decompressed many times.
> The reason we need mapred.map.output.compression.codec is for network traffic.
> We should not compress/decompress the data again and again during merge sort.
> We should only compress the final map output file that will be transmitted
> over the network.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
