[
https://issues.apache.org/jira/browse/HADOOP-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-3446:
----------------------------------
Attachment: 3446-2.patch
This changes reduce as follows:
* Instead of specifying {{fs.inmemory.size.mb}}, map outputs will consume
{{mapred.copy.inmem.percent}} of the maximum heap size as returned from
{{Runtime.maxMemory()}}, defaulting to 0.7. {{mapred.child.java.opts}} defaults
to 200mb and {{fs.inmemory.size.mb}} defaults to 75mb, so this might be
considered an incompatible change.
* The memory threshold at which the in-memory merge will start during the
shuffle is now user-configurable ({{mapred.inmem.merge.usage}}), defaulting to
the old value of 0.66. {{mapred.inmem.merge.threshold}} still controls the
maximum number of segments
* Instead of performing a final in-memory merge, the segments are left in
memory. At the beginning of the sort phase, the ReduceCopier is queried for an
Iterator to the reduce. A user-configurable property
{{mapred.reduce.inmem.percent}} determines the maximum size of the segments to
be merged from memory during the reduce, relative to the ShuffleRamManager
threshold. If the retained segments exceed this threshold, then they must be
written to disk before the reduce starts. If there sufficient segments already
on disk to require intermediate merges, they will be rolled into the first
merge, otherwise they will be merged to disk. The merge into the reduce will
contain all the segments that fit below the in-memory reduce threshold from RAM
and from the on-disk segments. So given:
{noformat}
+----+ <- Max heap memory (e.g. -Xmx512m) (H)
| |
|----| <- mapred.copy.inmem.percent (C)
| |
| |
|----| <- mapred.reduce.inmem.percent (R)
| |
+----+
{noformat}
The maximum memory used for copying map output wil be {{H*C}} while the minimum
memory available to the reduce will be {{H*(1-C*R)}}
This passes all unit tests on my machine. I'll work on measuring its
performance and post the results presently.
> The reduce task should not flush the in memory file system before starting
> the reducer
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-3446
> URL: https://issues.apache.org/jira/browse/HADOOP-3446
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Chris Douglas
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: 3446-0.patch, 3446-1.patch, 3446-2.patch
>
>
> In the case where the entire reduce inputs fit in ram, we currently force the
> input to disk and re-read it before giving it to the reducer. It would be
> much better if we merged from the ramfs and any spills to feed the reducer
> its input.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
