Hi Viswanathan, this looks like your job history is full, and is filling up your jobtracker heap:
> 2014-04-12 02:25:47,963 ERROR org.apache.hadoop.mapred.JobHistory: Unable to > move history file to DONE canonical subfolder. > java.lang.OutOfMemoryError: Java heap space Have you set any value for the following property, on your mapred-site.xml? mapred.jobtracker.completeuserjobs.maximum If you had not set this, jobtracker will use a default of 100, which may be too much depending on your heap size. If you had executed a lot of jobs since your last jobtracker restart, that may indicate this is the cause for this OOME (OutOfMemoryError). If so, try set the property mentioned to a smaller value, say “10”, then restart your jobtracker. Cheers, Wellington. On 12 Apr 2014, at 08:27, Viswanathan J <[email protected]> wrote: > Hi, > > I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without any > issues. > > Suddenly I got the below error in Jobtracker and jobs are failed to run. > > Is this issue in JT or TT or Jetty issue? > > 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION > java.lang.OutOfMemoryError: Java heap space > at java.util.HashMap.newKeyIterator(HashMap.java:840) > at java.util.HashMap$KeySet.iterator(HashMap.java:874) > at java.util.HashSet.iterator(HashSet.java:153) > at > sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) > at > org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) > at > org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) > at > org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > 2014-04-12 02:26:02,093 ERROR > org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException > as:root cause:java.io.IOException: java.lang.OutOfMemoryError: Java heap space > 2014-04-12 02:25:47,963 ERROR org.mortbay.log: Error for /jobtracker.jsp > java.lang.OutOfMemoryError: Java heap space > at java.lang.String.toCharArray(String.java:2725) > at sun.net.www.ParseUtil.encodePath(ParseUtil.java:86) > at > sun.misc.URLClassPath$JarLoader.checkResource(URLClassPath.java:674) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:759) > at sun.misc.URLClassPath.getResource(URLClassPath.java:169) > at java.net.URLClassLoader$1.run(URLClassLoader.java:194) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at > org.apache.hadoop.mapred.jobtracker_jsp._jspService(jobtracker_jsp.java:508) > at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > at > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:914) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > 2014-04-12 02:25:47,963 ERROR org.apache.hadoop.mapred.JobHistory: Unable to > move history file to DONE canonical subfolder. > java.lang.OutOfMemoryError: Java heap space > > Please help as I'm facing in production. > > -- > Regards, > Viswa.J
