Hi Ning, With hadoop-0.20.1 (apache release) on the UI (following the tracking URL) I only see a pending map task and when I click through fnally I see: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.mapred.JobInProgress.getTaskInProgress(JobInProgress.java:2523) at org.apache.hadoop.mapred.taskdetails_jsp._jspService(taskdetails_jsp.java:118) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
I feel this is not a root cause exception. So I have not been able to find the root cause exception anywhere on the jobtracker UI. I pasted what I found in /tmp/<username>/hive.log and that wasn't very indicative either. Pradeep ________________________________ From: Ning Zhang [mailto:nzh...@facebook.com] Sent: Tuesday, September 28, 2010 11:24 AM To: <hive-user@hadoop.apache.org> Subject: Re: Regression in trunk? (RE: Insert overwrite error using hive trunk) Prodeep, can you open the tracking URL printed out from the log and click through to the task log? The real error should be printed over there. The link may be expired so you need to rerun the query and click on the new one. I'm suspecting the error is due to the fact that CombineHiveInputFormat with the Hadoop version you are using. Again, the first thing is to check the task log through the tracking URL. Tracking URL = http://<hostname>:50030/jobdetails.jsp?jobid=job_201009251752_1341 On Sep 27, 2010, at 5:58 PM, Pradeep Kamath wrote: Here is some relevant stuff from /tmp/pradeepk/hive.logs - can't make much out of it: 2010-09-27 17:40:01,081 INFO exec.MapRedTask (SessionState.java:printInfo(268)) - Starting Job = job_201009251752_1341, Tracking URL = http://<hostname>:50030/jobdetails.jsp?jobid=job_201009251752_1341 2010-09-27 17:40:01,081 INFO exec.MapRedTask (SessionState.java:printInfo(268)) - Kill Command = /homes/pradeepk/hadoopcluster/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=<hostname>:50020 -kill job_201009251752_1341 2010-09-27 17:40:01,081 DEBUG ipc.Client (Client.java:sendParam(469)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk sending #129 2010-09-27 17:40:01,083 DEBUG ipc.Client (Client.java:receiveResponse(504)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk got value #129 2010-09-27 17:40:01,083 DEBUG ipc.RPC (RPC.java:invoke(225)) - Call: getJobStatus 2 2010-09-27 17:40:02,086 DEBUG ipc.Client (Client.java:sendParam(469)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk sending #130 2010-09-27 17:40:02,090 DEBUG ipc.Client (Client.java:receiveResponse(504)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk got value #130 2010-09-27 17:40:02,091 DEBUG ipc.RPC (RPC.java:invoke(225)) - Call: getJobStatus 5 2010-09-27 17:40:02,092 DEBUG ipc.Client (Client.java:sendParam(469)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk sending #131 2010-09-27 17:40:02,093 DEBUG ipc.Client (Client.java:receiveResponse(504)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk got value #131 2010-09-27 17:40:02,094 DEBUG ipc.RPC (RPC.java:invoke(225)) - Call: getJobStatus 2 2010-09-27 17:40:02,094 DEBUG ipc.Client (Client.java:sendParam(469)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk sending #132 2010-09-27 17:40:02,096 DEBUG ipc.Client (Client.java:receiveResponse(504)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk got value #132 2010-09-27 17:40:02,096 DEBUG ipc.RPC (RPC.java:invoke(225)) - Call: getJobProfile 2 2010-09-27 17:40:02,096 DEBUG ipc.Client (Client.java:sendParam(469)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk sending #133 2010-09-27 17:40:02,100 DEBUG ipc.Client (Client.java:receiveResponse(504)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk got value #133 2010-09-27 17:40:02,100 DEBUG ipc.RPC (RPC.java:invoke(225)) - Call: getJobCounters 4 2010-09-27 17:40:02,101 DEBUG mapred.Counters (Counters.java:<init>(151)) - Creating group org.apache.hadoop.hive.ql.exec.Operator$ProgressCounter with nothing 2010-09-27 17:40:02,101 DEBUG mapred.Counters (Counters.java:getCounterForName(277)) - Adding CREATED_FILES 2010-09-27 17:40:02,103 INFO exec.MapRedTask (SessionState.java:printInfo(268)) - 2010-09-27 17:40:02,103 Stage-2 map = 100%, reduce = 100% 2010-09-27 17:40:02,104 DEBUG ipc.Client (Client.java:sendParam(469)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk sending #134 2010-09-27 17:40:02,105 DEBUG ipc.Client (Client.java:receiveResponse(504)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk got value #134 2010-09-27 17:40:02,106 DEBUG ipc.RPC (RPC.java:invoke(225)) - Call: getJobStatus 2 2010-09-27 17:40:02,106 DEBUG ipc.Client (Client.java:sendParam(469)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk sending #135 2010-09-27 17:40:02,108 DEBUG ipc.Client (Client.java:receiveResponse(504)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk got value #135 2010-09-27 17:40:02,108 DEBUG ipc.RPC (RPC.java:invoke(225)) - Call: getJobCounters 2 2010-09-27 17:40:02,109 DEBUG mapred.Counters (Counters.java:<init>(151)) - Creating group org.apache.hadoop.hive.ql.exec.Operator$ProgressCounter with nothing 2010-09-27 17:40:02,109 DEBUG mapred.Counters (Counters.java:getCounterForName(277)) - Adding CREATED_FILES 2010-09-27 17:40:02,109 DEBUG ipc.Client (Client.java:sendParam(469)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk sending #136 2010-09-27 17:40:02,111 DEBUG ipc.Client (Client.java:receiveResponse(504)) - IPC Client (47) connection to <hostname>/216.252.118.203:50020 from pradeepk got value #136 2010-09-27 17:40:02,111 DEBUG ipc.RPC (RPC.java:invoke(225)) - Call: getJobStatus 2 2010-09-27 17:40:02,112 ERROR exec.MapRedTask (SessionState.java:printError(277)) - Ended Job = job_201009251752_1341 with errors Ning Zhang wrote: >From the error info, it seems the 2nd job has been launched and failed. So I'm >assuming there are map tasks started? If not, you can find the error message >in the client log file /tmp/<userid>/hive.log at the machine running hive >after setting the hive.root.logger property Steven mentioned. On Sep 27, 2010, at 1:11 PM, Steven Wong wrote: >>>>> 2010-09-24 13:29:03,504 Stage-2 map = 100%, reduce = 100% >>>>> >>>>> Ended Job = job_201009241059_0282 with errors >>>>> >>>>> FAILED: Execution Error, return code 2 from >>>>> org.apache.hadoop.hive.ql.exec.MapRedTask >>>>> >>>>>