[CONF] Apache Tomcat > OutOfMemory
Title: Message Title
There's 1 new edit on this page
OutOfMemory
Felix Schumacher edited this page
Here's the version comment
Felix Schumacher edited at 10:50 AM
Note about MetaSpae instead of PermGen
Here's what changed:
...
A servlet trying to load a several GBytes file into memory will surely kill the server. These kind of errors must be considered a simple bug in our program.
To compensate for the data your servlet tries to load, you increase the heap size so that there is no room to create the stack size for the threads that need to be created. The memory required by each thread will vary by OS but can be as high as 2M by default and in some OS's (like Debian Sarge) is not reducible with the -Xss parameter. Rule of Thumb, use no more than 1G for heap space in a 32-bit web application.
Deep recursive algorithms can also lead to Out Of Memory problems. In this case, the only fixes are increasing the thread stack size (-Xss), or refactoring the algorithms to reduce the depth, or the local data size per call.
A webapp that uses lots of libraries with many dependencies, or a server maintaining lots of webapps could exhauste the JVM PermGen space. This space is where the VM stores the classes and methods data. In those cases, the fix is to increase this size. The Sun VM has the flag -XX:MaxPermSize that allows to set its size (the default value is 64M)
Info
PermGen has been integrated into a new concept called MetaSpace from Java 8 on. The old setting will generate a warning and will be ignored by newer JVMs.
Hard references to classes can prevent the garbage collector from reclaiming the memory allocated for them when a ClassLoader is discarded. This will occur on JSP recompilations, and webapps reloads. If these operations are common in a webapp having these kinds of problems, it will be a matter of time, until the PermGen space gets full and an Out Of Memory is thrown.
... Any threads a web application starts, a web application should stop. ServletContextListener is your friend. Note Tomcat 7 will warn you if you do this and will also provide a (highly dangerous - use at your own risk) option to terminate the threads. DriverManager If you load a java.sql.Driver in your own classloader (or servlets), the driver should be removed before undeploying. Each driver is registered in DriverManager which is loaded in system classloader and references the local driver. Note Tomcat will do this for you if you forget.
No Format
Enumeration drivers = DriverManager.getDrivers();
ArrayList driversToUnload=new ArrayList();
while (drivers.hasMoreElements()) {
Driver driver = drivers.nextElement();
if (driver.getClass().getClassLoader().equals(getClass().getClassLoader())) {
driversToUnload.add(driver);
}
}
for (Driver driver : driversToUnload) {
DriverManager.deregisterDriver(driver);
}
ThreadLocal The lifecycle of a ThreadLocal should match that of a request. There is no guarantee that a thread will ever be used to process a request again so if a ThreadLocal is left on the thread at the end of the request there may be no opportunity for the web application to clean it up. Note Tomcat 7 will do this for you. ContextClassLoader There are various parts of the Java API that retain a permanent reference to the context class loader. If this happens to be a web application class loader then a memory leak will occur. Tomcat provides workarounds for these where known but there are undoubtedly others. ...
Go to page history
View page
Stop watching space
•
Manage notifications
This message was sent by Atlassian Confluence 6.15.2
[CONF] Apache Tomcat > OutOfMemory
Title: Message Title
There's 1 new edit on this page
OutOfMemory
Felix Schumacher edited this page
Here's the version comment
Felix Schumacher edited at 10:41 AM
Enable code macro to pretty print xml
Here's what changed:
...
A servlet trying to load a several GBytes file into memory will surely kill the server. These kind of errors must be considered a simple bug in our program.
To compensate for the data your servlet tries to load, you increase the heap size so that there is no room to create the stack size for the threads that need to be created. The memory required by each thread will vary by OS but can be as high as 2M by default and in some OS's (like Debian Sarge) is not reducible with the -Xss parameter. Rule of Thumb, use no more than 1G for heap space in a 32-bit web application.
Deep recursive algorithms can also lead to Out Of Memory problems. In this case, the only fixes are increasing the thread stack size (-Xss), or refactoring the algorithms to reduce the depth, or the local data size per call.
A webapp that uses lots of libraries with many dependencies, or a server maintaining lots of webapps could exhauste the JVM PermGen space. This space is where the VM stores the classes and methods data. In those cases, the fix is to increase this size. The Sun VM has the flag -XX:MaxPermSize that allows to set its size (the default value is 64M)
Hard references to classes can prevent the garbage collector from reclaiming the memory allocated for them when a ClassLoader is discarded. This will occur on JSP recompilations, and webapps reloads. If these operations are common in a webapp having these kinds of problems, it will be a matter of time, until the PermGen space gets full and an Out Of Memory is thrown.
... Any threads a web application starts, a web application should stop. ServletContextListener is your friend. Note Tomcat 7 will warn you if you do this and will also provide a (highly dangerous - use at your own risk) option to terminate the threads. DriverManager If you load a java.sql.Driver in your own classloader (or servlets), the driver should be removed before undeploying. Each driver is registered in DriverManager which is loaded in system classloader and references the local driver. Note Tomcat will do this for you if you forget.
No Format
Enumeration drivers = DriverManager.getDrivers();
ArrayList driversToUnload=new ArrayList();
while (drivers.hasMoreElements()) {
Driver driver = drivers.nextElement();
if (driver.getClass().getClassLoader().equals(getClass().getClassLoader())) {
driversToUnload.add(driver);
}
}
for (Driver driver : driversToUnload) {
DriverManager.deregisterDriver(driver);
}
ThreadLocal The lifecycle of a ThreadLocal should match that of a request. There is no guarantee that a thread will ever be used to process a request again so if a ThreadLocal is left on the thread at the end of the request there may be no opportunity for the web application to clean it up. Note Tomcat 7 will do this for you. ContextClassLoader There are various parts of the Java API that retain a permanent reference to the context class loader. If this happens to be a web application class loader then a memory leak will occur. Tomcat provides workarounds for these where known but there are undoubtedly others. ... Please remember that a JSP page, even one that simply prints out “OK”, will create a session. This is by design and if you do not want it to create a session you need to explicitly indicate that in your JSP. For example:
No Format
Code Block
language
php
<%@ page session="false" %>
This is important in scenarios where you are doing load testing and using custom HTTP clients, because these clients may not be handling sessions correctly and thus end up creating a new session every time they access the page. ... It is also possible to limit the number of active sessions by setting maxActiveSessions attribute on a Manager element, e.g.
No Format
Code Block
language
xml
Go to page history
View page
Stop watching space
•
Manage notifications
This message was sent by Atlassian Confluence 6.15.2
[CONF] Apache Tomcat > Troubleshooting and Diagnostics
Title: Message Title There's 2 new edits on this page Troubleshooting and Diagnostics Felix Schumacher edited this page Here's the version comments Felix Schumacher edited at 10:25 AM Correct Typo Felix Schumacher edited at 10:24 AM Updated Links to FAQs for Memory and Developing Here's what changed: ... How To: Capture a thread dump How To: Capture a heap dump How To: Examine a Stacktrace How To: Configure Tomcat for debugging FAQ: Developing FAQ: Memory Tomcat Memory Leak Protection Sun Technical Article: Monitoring and Managing Java SE 6 Platform Applications Notes on using JMX clients ... Look into Tomcat access log (the log file generated by AccessLogValve). If your request is not listed there, then it has not been processed by Tomcat. You need to look elsewhere (e.g. at your firewall). You will see what IP address your client is using, and whether it is using an IPv4 (127.0.0.1) or IPv6 address (0:0:0:0:0:0:0:1). Modern operating systems can use IPv6 addresses for localhost / local network access, while external network is still using IPv4. 2. Take a thread dump. This way you will find out what Tomcat is actually doing. If you are troubleshooting some process that takes noticeable time, take several (three) thread dumps with some interval between them. This way you will see if there are any changes, any progress. 3. Try debugging. A good place for a breakpoint is org.apache.catalina.connector.CoyoteAdapter.service() method. That is the entry point from Tomcat connectors and into the Servlet engine. At that place your request has already been received and its processing starts. ... You can also search the archives of the Tomcat users' mailing lists for previous discussions mentioning the RECYCLE_FACADES flag. 2. Read about Java ImageIO issue. Accessing response objects after their lifetime can lead to security issues in your application, such as sending responses to wrong clients, mixing up responses. If you can reproduce the issue and the above diagnostic does not show your own bug, but a bug in Apache Tomcat, ... Go to page history View page Stop watching space • Manage notifications This message was sent by Atlassian Confluence 6.15.2
[CONF] Apache Tomcat > Developing
Title: Message Title There's 1 new edit on this page Developing Felix Schumacher edited this page Here's the version comment Felix Schumacher edited at 10:28 AM Use git instead of svn Here's what changed: ... How do I start hacking Tomcat in Eclipse? Briefly: No Format $ svngit checkoutclone httphttps://svngithub.com/apache.org/repos/asf/tomcat/trunktomcat.git (or whatever branch you want: clearly, this would be better to do directly from within Eclipse but it's easier to describe as a command) $ cd trunktomcat $ echo "base.path=/path/to/where/tomcat/can/put/its/3rd-party/libs" > build.properties $ ant ide-eclipse ... How do I remotely debug Tomcat using NetBeans? This answer assumes that you know how to work with a NetBeans Project, and also how to use the NetBeans debugger. If not, please go to http://www.netbeans.org/kb/using-netbeans/40/debug.html and read up on how to use NetBeans and its debugger. ... Go to page history View page Stop watching space • Manage notifications This message was sent by Atlassian Confluence 6.15.2
[CONF] Apache Tomcat > Troubleshooting and Diagnostics
Title: Message Title
There's 1 new edit on this page
Troubleshooting and Diagnostics
Felix Schumacher edited this page
Here's what changed:
...
How To: Capture a thread dump
How To: Capture a heap dump
How To: Examine a Stacktrace
How To: Configure Tomcat for debugging
FAQ: Developing
FAQ: Memory
Tomcat Memory Leak Protection
Sun Technical Article: Monitoring and Managing Java SE 6 Platform Applications
Notes on using JMX clients
...
jinfo - Prints JVM process info
jstack - Prints thread stack traces
jmap - Dumps heap and shows heap status
jhat - Heap Analyzer Tool
jcmd - Multitool intended to replace the above JDK tools
Profilers & Heap Analyzers
Eclipse Memory Analyzer (MAT)
YourKit Profiler
VisualVM Docs
Anchor
usingjmxclients
usingjmxclients
...
Look into Tomcat access log (the log file generated by AccessLogValve).
If your request is not listed there, then it has not been processed by Tomcat. You need to look elsewhere (e.g. at your firewall).
You will see what IP address your client is using, and whether it is using an IPv4 (127.0.0.1) or IPv6 address (0:0:0:0:0:0:0:1). Modern operating systems can use IPv6 addresses for localhost / local network access, while external network is still using IPv4. 2. Take a thread dump. This way you will find out what Tomcat is actually doing.
If you are troubleshooting some process that takes noticeable time, take several (three) thread dumps with some interval between them. This way you will see if there are any changes, any progress. 3. Try debugging.
A good place for a breakpoint is org.apache.catalina.connector.CoyoteAdapter.service() method. That is the entry point from Tomcat connectors and into the Servlet engine. At that place your request has already been received and its processing starts.
... The main suspect is your own web application keeping a reference to Request / Response objects outside of their life cycle. The lifetime of the Response object is documented in the Servlet specification. Quoting from section "5.8 Lifetime of the Response Object" of Servlet 4.0 specification: "Each response object is valid only within the scope of a servlet’s service method, or within the scope of a filter’s doFilter method, unless the associated request object has asynchronous processing enabled for the component. If asynchronous processing on the associated request is started, then the response object remains valid until complete method on AsyncContext is called." In case of asynchronous processing, when an error occurs Tomcat notifies all registered AsyncListener}}s and then calls {{complete() automatically if none of the listeners have called it yet. (Reference: 61768) Also see sections "2.3.3.4 Thread Safety" and "3.13 Lifetime of the Request Object" of the same specification. To troubleshoot the issue: ... You can also search the archives of the Tomcat users' mailing lists for previous discussions mentioning the RECYCLE_FACADES flag. 2. Read about Java ImageIO issue. Accessing response objects after their lifetime can lead to security issues in your application, such as sending responses to wrong clients, mixing up responses. If you can reproduce the issue and the above diagnostic does not show your own bug, but a bug in Apache Tomcat, ...
Go to page history
View page
Stop watching space
•
Manage notifications
This message was sent by Atlassian Confluence 6.15.2
