[
https://issues.apache.org/jira/browse/MAPREDUCE-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-2910:
----------------------------------------
Fix Version/s: (was: 0.24.0)
> Allow empty MapOutputFile segments
> ----------------------------------
>
> Key: MAPREDUCE-2910
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2910
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task, tasktracker
> Affects Versions: 0.20.2, 0.23.0
> Reporter: Binglin Chang
> Priority: Minor
>
> As the scale of cluster and job get larger, we see a lot of empty partitions
> in MapOutputFile due to large reduce numbers or partition skew. When map
> output compression is enabled, empty map output partitions gets larger & has
> additional compressor/decompressor initialization overhead.
> This can be optimized by allowing empty MapOutputFile segments, where the
> rawLength & partLength of IndexRecord all equal to 0. Corresponding support
> need to be added to IFile reader, writer, and reduce shuffle copier.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
