[
https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-5251:
--------------------------------------
Target Version/s: 0.23.8, 2.0.4-alpha
Status: Patch Available (was: Open)
The issue here is was that any IOException that arises from
Fetcher.copyMapOutput is always blamed on map causing the map to be re-run.
But we can get an exception like DiskErrorException which is due to lack of
disk space on the reduce node and has nothing to do with map. The map task
should not be re-executed in this case.
To solve this,I've added code to catch this DiskErrorException,blame the reduce
and report this to the shuffle scheduler ,which in turn kills this reduce task
and would hopefully re-run it on another node which has disk space.
I manually tested this by changing the source and stubbing it to reserve 1
Peta-byte during the shuffle phase. This throws the DiskErrorException as
expected,blames the reduce and most importantly doesn't run the map task again.
I've also added a unit test to simulate this scenario.
> Reducer should not implicate map attempt if it has insufficient space to
> fetch map output
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5251
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.4-alpha, 0.23.7
> Reporter: Jason Lowe
> Assignee: Ashwin Shankar
> Attachments: MAPREDUCE-5251-2.txt
>
>
> A job can fail if a reducer happens to run on a node with insufficient space
> to hold a map attempt's output. The reducer keeps reporting the map attempt
> as bad, and if the map attempt ends up being re-launched too many times
> before the reducer decides maybe it is the real problem the job can fail.
> In that scenario it would be better to re-launch the reduce attempt and
> hopefully it will run on another node that has sufficient space to complete
> the shuffle. Reporting the map attempt is bad and relaunching the map task
> doesn't change the fact that the reducer can't hold the output.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
