Hi Sir, I have 4 node hadoop cluster with one of the node acting as master as well as slave node. The SecondayNameNode is running on one of the slave node. I am using 8 map (mapred.map.tasks) and 4 reduce (mapred.reduce.tasks) tasks. I am using the cluster for Nutch 1.4 crawling. I observed that, one of the slave node (node with secondary Name node) got down with below Exception in tasktracker log,
2012-09-21 00:37:40,819 WARN mapred.TaskTracker -
getMapOutput(attempt_201209201246_0063_m_000036_0,2) failed :
org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:550)
at
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:571)
at
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1005)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:648)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:579)
at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2940)
....
2012-09-21 00:37:40,820 WARN mortbay.log - Committed before 410
getMapOutput(attempt_201209201246_0063_m_000036_0,2) failed :
org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:550)
at
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:571)
at
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1005)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:648)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:579)
at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2940)
....
2012-09-21 00:37:40,823 ERROR mortbay.log - /mapOutput
java.lang.IllegalStateException: Committed
at org.mortbay.jetty.Response.resetBuffer(Response.java:1024)
at org.mortbay.jetty.Response.sendError(Response.java:240)
at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2963)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
I suspect, above Exceptions has caused the slave node to get down.
I found one reference on JIRA with such issue, but didn't get the root cause
and way to resolve such issues.
https://issues.apache.org/jira/browse/MAPREDUCE-5
Could you help me to understand the root cause of above exceptions and whether
this may cause the Tasktracker to get down?
Your comments will be helpful to understand and give hint to improve the hadoop
performance.
Thanks,
Yogesh Sangvikar
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the
property of Persistent Systems Ltd. It is intended only for the use of the
individual or entity to which it is addressed. If you are not the intended
recipient, you are not authorized to read, retain, copy, print, distribute or
use this message. If you have received this communication in error, please
notify the sender and delete all copies of this message. Persistent Systems
Ltd. does not accept any liability for virus infected mails.
hadoop-root-tasktracker-goicrawlslave01.nic.in.log
Description: hadoop-root-tasktracker-goicrawlslave01.nic.in.log
