[
https://issues.apache.org/jira/browse/MAPREDUCE-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Age Mooij updated MAPREDUCE-2088:
---------------------------------
Attachment: MAPREDUCE-2088-JobTracker.png
MAPREDUCE-2088-ReducerProgress.png
I attached two screenshots, one for the job overview as it currently stands and
one for the reducers progress page as it currently stands.
> TaskGraphServlet throws ArrayIndexOutOfBoundsException when progress % > 100%
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-2088
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2088
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Environment: Hadoop 0.20.2, r911707 running on a linux cluster with 4
> worker nodes and LZO compressed map output
> Reporter: Age Mooij
> Attachments: MAPREDUCE-2088-JobTracker.png,
> MAPREDUCE-2088-ReducerProgress.png
>
>
> I'm currently running a map reduce job in which the reducer inserts data into
> HBase. It is a long running job dealing with hundreds of GB of map output
> data and 9 out of 10 reducers are currently showing 110+% progress (i.e. the
> total job reduce progress is 100% but clickig through to the reducer list
> shows reducers with percentages over 100%).
> The number of reduce input records is 4,299,991,005
> While keeping an eye on it through the job detail page of the JobTracker web
> gui, the Jobtracker started generating exceptions every 30 seconds which
> turned out to be generated by the TaskGrapServlet that displays reducer
> progress. The stacktrace also showed up in the web gui in the area that
> normally holds the reducer progress graph. 30 seconds is of course the
> refresh rate of the status page.
> I guess the root cause here is progress percentages being greater than 100%.
> I found issue HADOOP-5210 which reports a cause for progress crossing 100%
> but it is marked as fixed in 0.20.1 and I'm running 0.20.2.
> One thing that might impact the progress percentages is the fact that we LZO
> compress our map outputs. For the above job uncompressed output ("Map output
> bytes") was 2,462,412,228,874 which compressed to 694,405,054,632
> (FILE_BYTES_WRITTEN/map).
> Here's the exception:
> 2010-09-23 18:03:05,484 ERROR org.mortbay.log: /taskgraph
> java.lang.ArrayIndexOutOfBoundsException: 3
> at
> org.apache.hadoop.mapred.TaskGraphServlet.getReduceAvarageProgresses(TaskGraphServlet.java:199)
> at
> org.apache.hadoop.mapred.TaskGraphServlet.doGet(TaskGraphServlet.java:131)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
