[
https://issues.apache.org/jira/browse/MAPREDUCE-26?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-26.
------------------------------
Resolution: Not A Problem
We attempt a proper rename now, and only fall back to copy+delete otherwise.
> The shuffle keeps the ReduceTask locked while doing a FileSystem.rename
> leading to task timeouts
> ------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-26
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-26
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> The shuffle in ReduceTask.ReduceCopier.MapOutputCopier.copyOutput locks the
> entire ReduceTask while doing a FileSystem.rename operation. Unfortunately
> the RawLocalFileSystem implements rename as a copy and delete, which can take
> a long time. As a result the reduce is being killed as not reporting progress
> for 10 minutes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
