[
https://issues.apache.org/jira/browse/HADOOP-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710602#action_12710602
]
Jothi Padmanabhan commented on HADOOP-5865:
-------------------------------------------
The Jetty EOFException is observed when the Reduce Task prematurely closes a
connection to a jetty server. The RT might decide to do this when the expected
map output size is less than its memory cache limit, but cannot fetch it now
because it has already fetched several other map outputs to memory. So, it
triggers a flush of the existing map outputs from memory to disk and once this
flush is complete, it will attempt to fetch the map output again by reopening
the connection to the Jetty. In short, it is OK to see such messages.
This should neither cause a task nor a job failure. Are you seeing failures?
The reason why these error messages appear for some jobs and not for others is
primarily got to do with the size of map outputs and the heap size allocated
for the reducer.
> Shuffle's getMapOutput() fails with EofException, followed by
> IllegalStateException
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-5865
> URL: https://issues.apache.org/jira/browse/HADOOP-5865
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Environment: Sun Java 1.6.0_13, OpenSolaris, running on a SunFire
> 4150 (x64) 10 node cluster
> Reporter: George Porter
>
> During the shuffle phase, I'm seeing a large sequence of the following
> actions:
> 1) WARN org.apache.hadoop.mapred.TaskTracker:
> getMapOutput(attempt_200905181452_0002_m_000010_0,0) failed :
> org.mortbay.jetty.EofException
> 2) WARN org.mortbay.log: Committed before 410
> getMapOutput(attempt_200905181452_0002_m_000010_0,0) failed :
> org.mortbay.jetty.EofException
> 3) ERROR org.mortbay.log: /mapOutput java.lang.IllegalStateException:
> Committed
> The map phase completes with 100%, and then the reduce phase crawls along
> with the above errors in each of the TaskTracker logs. None of the
> tasktrackers get lost. When I run non-data jobs like the 'pi' test from the
> example jar, everything works fine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
