[
https://issues.apache.org/jira/browse/MAPREDUCE-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035585#comment-13035585
]
Chris Douglas commented on MAPREDUCE-2510:
------------------------------------------
bq. Is there an issue about upgrading Jetty to 6.1.26?
None that I'm aware of. Upgrading from Jetty5 to Jetty6 was painful, but
upgrading within Jetty6 has been very easy.
bq. Why Jetty 6.1.26 does not have this behavior?
This showed up in a cluster while I was on call. The heap dump of the
TaskTracker showed exactly what you found, i.e. a large number of Runnable
instances in an array. Looking at the 6.1.14 source, it was obvious that the
circular buffer was failing to free references to requests queued in this
buffer, which expanded whenever its throughput lagged behind the request rate.
Looking at the 6.1.25 source, references to requests in this buffer were
correctly set to {{null}} after dispatching them. I don't know when this was
fixed, only that we've been running 6.1.26 and it's the smoothest version of
Jetty we've deployed.
> TaskTracker throw OutOfMemoryError after upgrade to jetty6
> ----------------------------------------------------------
>
> Key: MAPREDUCE-2510
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2510
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Liyin Liang
>
> Our product cluster's TaskTracker sometimes throw OutOfMemoryError after
> upgrade to jetty6. The exception in TT's log is as follows:
> 2011-05-17 19:16:40,756 ERROR org.mortbay.log: Error for /mapOutput
> java.lang.OutOfMemoryError: Java heap space
> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
> at
> org.apache.hadoop.fs.BufferedFSInputStream.<init>(BufferedFSInputStream.java:44)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:176)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:359)
> at
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3040)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> Exceptions in .out file:
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "process reaper" java.lang.OutOfMemoryError: Java heap
> space
> Exception in thread "pool-1-thread-1" java.lang.OutOfMemoryError: Java heap
> space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.reflect.InvocationTargetException
> Exception in thread "IPC Server handler 6 on 50050" at
> sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:126)
> at org.mortbay.log.Log.warn(Log.java:181)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:449)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
