[
https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated MAPREDUCE-6724:
----------------------------------
Attachment: mapreduce6724.001.patch
Thanks [~jira.shegalov] a lot for reviewing the approach proposed. Uploading a
patch that implements it.
1) The patch caps maxSingleShuffleLimit at Integer.MAX_VALUE, so that all
shuffle with output bigger than Integer.MAX_VALUE will be done on disk,
avoiding the unsafe conversion
2) The reset of the patch is just refactoring MergeManagerImpl so that a
no-merge MergeManager can be created in the new test.
> Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-6724
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Attachments: mapreduce6724.001.patch
>
>
> When shuffle is done in memory, MergeManagerImpl converts the requested size
> to an int to allocate an instance of InMemoryMapOutput. This results in an
> overflow if the requested size is bigger than Integer.MAX_VALUE and
> eventually causes the reducer to fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
