Hi there,
Howdy. I observe at times that few of the reduce tasks
hangs during copy phase and does not result in
failures also. Hence these tasks never complete nor
rerun for timeouts.
reduce > copy (1510 of 1540 at 1.57 MB/s) >
At the same time, I see that Jetty is out of threads
in its thread pool. Dont know if these 2 are related.
I also see the following exception for many of the MR
operations.
24 Feb 2007 02:52:14,438 WARN - No thread for
Socket[addr=/10.163.63.137,port=56019,localport=50060]
- at
org.mortbay.util.ThreadPool.run(ThreadPool.java:373)
24 Feb 2007 02:52:24,440 WARN - No thread for
Socket[addr=/10.163.63.137,port=56023,localport=50060]
- at
org.mortbay.util.ThreadPool.run(ThreadPool.java:373)
24 Feb 2007 02:53:01,582 WARN -
getMapOutput(task_0005_m_000432_0,30) failed :
java.net.SocketException: Connection reset
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.mortbay.http.ChunkingOutputStream.bypassWrite(ChunkingOutputStream.java:151)
at
org.mortbay.http.BufferedOutputStream.write(BufferedOutputStream.java:139)
at
org.mortbay.http.HttpOutputStream.write(HttpOutputStream.java:423)
at
org.mortbay.jetty.servlet.ServletOut.write(ServletOut.java:54)
at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1574)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
at
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
at
org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
at
org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at
org.mortbay.http.HttpServer.service(HttpServer.java:954)
at
org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at
org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
- at
org.apache.hadoop.mapred.TaskTracker.doGet(TaskTracker.java:1600)
24 Feb 2007 02:53:01,583 WARN -
getMapOutput(task_0005_m_001465_0,55) failed :
Has anyone experienced this? Any thoughts are greatly
appreciated.
Thanks,
Venkat
____________________________________________________________________________________
Don't pick lemons.
See all the new 2007 cars at Yahoo! Autos.
http://autos.yahoo.com/new_cars.html
