[
https://issues.apache.org/jira/browse/HADOOP-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amar Kamat updated HADOOP-5746:
-------------------------------
Attachment: 5746-testcase.patch
5746-reproduce.1.patch
5746.6.patch
Attaching the patch [5746.6.patch]. It looks like its a very timing issue to
reproduce/test this bug. The problem occurs when the pipe process finishes off
and then the output thread cranks up with some exception (like fs errors).
Attaching a framework change [5746-reproduce.1.patch] and a testcase
[/5746-testcase.patch] to verify the fix. The caller of
{{PipeMapRed.waitOutputThreads()}} i.e {{PipeMapRed.mapRedFinished()}} simply
ignores IOException hence changed the exception to {{RuntimeException}}. I dont
know why PipeMapRed.mapRedFinished() ignores IOException. But for now I have
kept it as it is.
> Errors encountered in MROutputThread after the last map/reduce call can go
> undetected
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-5746
> URL: https://issues.apache.org/jira/browse/HADOOP-5746
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Devaraj Das
> Fix For: 0.21.0
>
> Attachments: 5746-reproduce.1.patch, 5746-testcase.patch,
> 5746.1.patch, 5746.6.patch
>
>
> The framework map/reduce bridge methods make a check at the beginning of the
> respective methods whether _MROutputThread_ encountered an exception while
> writing keys/values that the streaming process emitted. However, if the
> exception happens in _MROutputThread_ after the last call to the map/reduce
> method, the exception goes undetected. An example of such an exception is an
> exception from the _DFSClient_ that fails to write to a file on the HDFS.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
