Anything in TaskTracker logs ? On Jan 31, 2012, at 10:18 AM, Markus Jelsma wrote:
> In our case, which seems to be the same problem, the web UI does not show > anything useful except the first line of the stack trace: > > 2012-01-03 21:16:27,256 WARN org.apache.hadoop.mapred.TaskLog: Failed to > retrieve stdout log for task: attempt_201201031651_0008_m_000233_0 > > Only the task tracker log shows a full stack trace. This happened on 1.0.0 > and > 0.20.205.0 but not 0.20.203.0. > > 2012-01-03 21:16:27,256 WARN org.apache.hadoop.mapred.TaskLog: Failed to > retrieve stdout log for task: attempt_201201031651_0008_m_000233_0 > java.io.FileNotFoundException: > /opt/hadoop/hadoop-0.20.205.0/libexec/../logs/userlogs/job_201201031651_0008/attempt_201201031651_0008_m_000233_0/log.index > (No such file or directory) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(SecureIOUtils.java:102) > at > org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:187) > at org.apache.hadoop.mapred.TaskLog$Reader.(TaskLogServlet.java:81) > at > org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > at > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > > >> Actually, all that is telling you is that the task failed and the >> job-client couldn't display the logs. >> >> Can you check the JT web-ui and see why the task failed ? >> >> If you don't see anything there, you can try see the TaskTracker logs on >> the node on which the task ran. >> >> Arun >> >> On Jan 31, 2012, at 3:21 AM, Marcin Cylke wrote: >>> Hi >>> >>> I've upgraded my hadoop cluster to version 1.0.0. The upgrade process >>> went relatively smoothly but it rendered the cluster inoperable due to >>> errors in jobtrackers operation: >>> >>> # in job output >>> Error reading task >>> outputhttp://hadoop4:50060/tasklog?plaintext=true&attemptid=attempt_20120 >>> 1311241_0003_m_000004_2&filter=stdout >>> >>> # in each of the jobtrackers' logs >>> WARN org.apache.hadoop.mapred.TaskLog: Failed to retrieve stderr log for >>> task: attempt_201201311241_0003_r_000000_1 >>> java.io.FileNotFoundException: >>> /usr/lib/hadoop-1.0.0/libexec/../logs/userlogs/job_201201311241_0003/atte >>> mpt_201201311241_0003_r_000000_1/log.index (No such file or directory) >>> >>> at java.io.FileInputStream.open(Native Method) >>> >>> These errors seem related to this two problems: >>> >>> http://grokbase.com/t/hadoop.apache.org/mapreduce-user/2012/01/error-read >>> ing-task-output-and-log-filenotfoundexceptions/03mjwctewcnxlgp2jkcrhvsgep >>> 4e >>> >>> https://issues.apache.org/jira/browse/MAPREDUCE-2846 >>> >>> But I've looked into the source code and the fix from MAPREDUCE-2846 is >>> there. Perhaps there is some other reason? >>> >>> Regards >>> Marcin >> >> -- >> Arun C. Murthy >> Hortonworks Inc. >> http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
