[
https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594124#comment-13594124
]
Ravi Prakash commented on MAPREDUCE-3685:
-----------------------------------------
This is probably more for just my reference than anything. Here's my
understanding from reading the code. This is very approximate and may be
inaccurate for some cases
IntermediateMemoryToMemoryMerger - Can be toggled on / off
- Merges map outputs *from* memory *to* memory
- When is it triggered? (If at all enabled, which it isn't by default) When the
number of in memory map outputs > memToMemMergeOutputsThreshold
I am guessing this was put in on the premise that it might be faster to sort
fewer number of streams even in memory. And also we can sort perhaps while
waiting to fetch.
InMemoryMerger
- Merges map outputs *from* memory *to* disk
- When is it triggered? When storing more map outputs in memory would cause to
go over memory allocated for shuffle.
> There are some bugs in implementation of MergeManager
> -----------------------------------------------------
>
> Key: MAPREDUCE-3685
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: anty.rao
> Assignee: anty
> Priority: Critical
> Attachments: MAPREDUCE-3685-branch-0.23.1.patch,
> MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch,
> MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch,
> MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch,
> MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch,
> MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch,
> MAPREDUCE-3685.patch, MAPREDUCE-3685.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
