[
https://issues.apache.org/jira/browse/HADOOP-2438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562157#action_12562157
]
Runping Qi commented on HADOOP-2438:
------------------------------------
If the stream mapper stalled for some reason and cannot consume the std input,
while the
Java MapRed wrapper continues to pipe to the mapper, then maybe too much data
will be
accumulated in the std input pipe.
That may cause broken pipe or oom exception. What did the mapper do?
> In streaming, jobs that used to work, crash in the map phase -- even if the
> mapper is /bin/cat
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-2438
> URL: https://issues.apache.org/jira/browse/HADOOP-2438
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.1
> Reporter: arkady borkovsky
>
> The exception is either "out of memory" of or "broken pipe" -- see both stack
> dumps bellow.
> st Hadoop input: |null|
> last tool output: |[EMAIL PROTECTED]|
> Date: Sat Dec 15 21:02:18 UTC 2007
> java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:96)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)
> at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:107)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)
> -------------------------------------------------
> java.io.IOException: MROutput/MRErrThread
> failed:java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect
> (MapTask.java:347)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run
> (PipeMapRed.java:344)
> at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:76)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:
> 1760)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
