[CONF] Apache Tomcat > Troubleshooting and Diagnostics

[Konstantin Kolinko (Confluence)]

Title: Message Title



 
 
 
There's 1 new edit on this page 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Troubleshooting and Diagnostics 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
Konstantin Kolinko edited this page 
 
 
  
 
 

 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
Here's the version comment 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
Konstantin Kolinko edited at 10:21 AM 
 
 
  
 
 

 
 
 
 
 
 
 
 
 Add an item about "File leak detector" (moving it from a top-level "Debugging" page)  
 
 
  
 
 
  
 
 

 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Here's what changed: 
 
 
 
 
 
 
 
 
 
 
 ... 
 
 How To: Capture a thread dump  
 How To: Capture a heap dump  
 How To: Examine a Stacktrace  
 How To: Configure Tomcat for debugging  
 FAQ: Developing  
 FAQ: Memory  
 Tomcat Memory Leak Protection  
 Notes on using JMX clients  
 Tools JMX Clients 
 
JJConsole: Documentation  
VisualVM: Documentation, Project  
 ... Accessing response objects after their lifetime can lead to security issues in your application, such as sending responses to wrong clients, mixing up responses. If you can reproduce the issue and the above diagnostic does not show your own bug, but a bug in Apache Tomcat, if the problem manifests as a security issue, see how to report it.  Troubleshooting "Too many open file descriptors"   The code that opens the descriptors can be identified using a tool such as http://file-leak-detector.kohsuke.org/   
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Go to page history 
 
 
  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
View page 
 
 
  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
Stop watching space
• 
 
 
 
 
 
 
Manage notifications 
 
 
 
 
 
 
 
 
 
 
  
 
 
This message was sent by Atlassian Confluence 7.1.2  
 
 
  
 
 
 
 
 
 
 
 
 

[CONF] Apache Tomcat > Troubleshooting and Diagnostics

[Konstantin Kolinko (Confluence)]

Title: Message Title



 
 
 
There's 1 new edit on this page 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Troubleshooting and Diagnostics 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
Konstantin Kolinko edited this page 
 
 
  
 
 

 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
Here's the version comment 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
Konstantin Kolinko edited at 02:54 AM 
 
 
  
 
 

 
 
 
 
 
 
 
 
 Improve formatting of Table of Contents.  
 
 
  
 
 
  
 
 

 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Here's what changed: 
 
 
 
 
 
 
 
 
 
 
  Troubleshooting and Diagnostics techniques.   Permalink to this page: https://cwiki.apache.org/confluence/x/yColBg   Troubleshooting and Diagnostics techniques.   Table of Contents   
 
 
 
 Table of Contents 
 
 
 
 
 
 
 
 
exclude 
Table of Contents 
 
 
  
 
 
  Techniques & Reference 
 
 How To: Capture a thread dump  
 How To: Capture a heap dump  
 How To: Examine a Stacktrace  
 How To: Configure Tomcat for debugging  
 FAQ: Developing  
 FAQ: Memory  
 Tomcat Memory Leak Protection  
 Notes on using JMX clients  
 ...  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Go to page history 
 
 
  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
View page 
 
 
  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
Stop watching space
• 
 
 
 
 
 
 
Manage notifications 
 
 
 
 
 
 
 
 
 
 
  
 
 
This message was sent by Atlassian Confluence 6.15.8  
 
 
  
 
 
 
 
 
 
 
 
 

[CONF] Apache Tomcat > Troubleshooting and Diagnostics

[Konstantin Kolinko (Confluence)]

Title: Message Title



 
 
 
There's 1 new edit on this page 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Troubleshooting and Diagnostics 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
Konstantin Kolinko edited this page 
 
 
  
 
 

 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
Here's the version comment 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
Konstantin Kolinko edited at 01:16 AM 
 
 
  
 
 

 
 
 
 
 
 
 
 
 Add a permalink. Fix formatting. Convert links to https.  
 
 
  
 
 
  
 
 

 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Here's what changed: 
 
 
 
 
 
 
 
 
 
 
 Troubleshooting and Diagnostics techniques.  Permalink to this page: https://cwiki.apache.org/confluence/x/yColBg   
 
 
 
 Table of Contents 
 
 
  Techniques & Reference 
 
 How To: Capture a thread dump  
 How To: Capture a heap dump  
 How To: Examine a Stacktrace  
 How To: Configure Tomcat for debugging  
 FAQ: Developing  
 FAQ: Memory  
 Tomcat Memory Leak Protection  
 Sun Technical Article: Monitoring and Managing Java SE 6 Platform Applications  
 Notes on using JMX clients  
 Tools JMX Clients 
 
 JConsole DocsJJConsole: Documentation  
 VisualVM Docs: Documentation, Project  
  JDK tools  ...  
 
 jinfo - Prints JVM process info  
 jstack - Prints thread stack traces  
 jmap - Dumps heap and shows heap status  
 jhat - Heap Analyzer Tool  
 jcmd - Multitool intended to replace the above JDK tools  
 Profilers & Heap Analyzers 
 
 Eclipse Memory Analyzer (MAT)  
 YourKit Profiler  
  VisualVM Docs   
  
 
 
 
 Anchor 
 
 
 
 
 
 
 
 
 
usingjmxclients 
 
 
 
usingjmxclients 
 
 
  
 
 
  ... NB(2) On Windows, if Tomcat is started using a service wrapper, this will prevent JConsole & VisualVM from using the local JMX connector stub.  Java 5 JConsole and Remote Management FAQ  From Java 6 onward a process needn't does not need to have the management agent enabled when it starts, as the Attach API permits the management agent to be activated on demand. ... 
 
Look into Tomcat access log (the log file generated by AccessLogValve).  
 
 If your request is not listed there, then it has not been processed by Tomcat. You need to look elsewhere (e.g. at your firewall).  
 You will see what IP address your client is using, and whether it is using an IPv4 (127.0.0.1) or IPv6 address (0:0:0:0:0:0:0:1).  Modern operating systems can use IPv6 addresses for localhost / local network access, while external network is still using IPv4.  
 2.   
 
 
 Take a thread dump. This way you will find out what Tomcat is actually doing. 
 
 If you are troubleshooting some process that takes noticeable time, take several (three) thread dumps with some interval between them. This way you will see if there are any changes, any progress.  
 3.   
 Try debugging. 
 
 A good place for a breakpoint is org.apache.catalina.connector.CoyoteAdapter.service() method. That is the entry point from Tomcat connectors and into the Servlet engine. At that place your request has already been received and its processing starts.  
  
 Troubleshooting unexpected Response state problems ... The lifetime of the Response object is documented in the Servlet specification. Quoting from section "5.8 Lifetime of the Response Object" of Servlet 4.0 specification:  "Each response object is valid only within the scope of a servlet’s service method, or within the scope of a filter’s doFilter method, unless the associated request object has asynchronous processing enabled for the component. If asynchronous processing on the associated request is started, then the response object remains valid until complete method on AsyncContext is called."  In case of asynchronous processing, when an error occurs Tomcat notifies all registered AsyncListener}}s and then calls  {{complete() automatically  automatically if none of the listeners have called it yet. (Reference: 61768) ... To troubleshoot the issue: 
 
Set the following system property in Tomcat configuration:  org.apache.catalina.connector.RECYCLE_FACADES=true  When the above flag is set, Tomcat recycles facades to its internal objects when request processing completes. This makes it easier to spot illegal access when it happens, instead of waiting until side effects of such access become visible.  This flag is also mentioned on the Security Considerations page.   
 ... 
 
  The flag is true when Tomcat runs with enabled Java Security Manager.   You can also search the archives of the Tomcat users' mailing lists for previous discussions mentioning the RECYCLE_FACADES flag.
 ... 
 
   
 Read about Java ImageIO issue.  
 Accessing response objects after their lifetime can lead to security issues in your application, such as sending responses to wrong clients, mixing up responses. If you can reproduce the issue and the above diagnostic does not show your own bug, but a bug in Apache Tomcat,  if the problem manifests as a security issue, see how to report it.  
 
 
  
 
 
  
 

Re: [CONF] Apache Tomcat > Troubleshooting and Diagnostics

[Felix Schumacher]

Am 05.06.19 um 22:13 schrieb Eugène Adell:
> Hello,
>
> if I may suggest something, over the years I have found very useful
> jstat and GCViewer for detecting GC suspicious behaviors. Both are of
> course free, the first one giving information on a live JVM, and the
> second being more interesting for an offline analysis (though it can
> be updated automatically). With that you can see an OOM coming, and
> you also can have some clues for the heap settings if you want to tune
> them. You also have a visual proof when you perform a major upgrade
> and want to see if it had an impact on memory, by comparing 2 or 3
> pictures before with 2 or 3 pictures after. They were not very often
> mentionned in the users list, maybe they are underrated ?

I like gc logs a lot, as they give me a quick picture of the used
memory. I have configured all my tomcat instances to log there gc's. And
my first go to when an instance feels slow is to use jstack and look at
the fgc count.

So feel free to add a paragraph about using gc logs and corresponding
tools to the wiki.

Felix

>
> best regards
> E.A.
>
> Le mar. 4 juin 2019 à 21:00, Felix Schumacher (Confluence)
> mailto:no-re...@apache.org>> a écrit :
>
> There's *1 new edit* on this page
>  
> page icon
> 
> 
>
>
>   Troubleshooting and Diagnostics
> 
> 
>
>
>   
> Felix Schumacher edited this page
>
>
>   
>
> Here's what changed:
>
> ...
>
>   * How To: Capture a thread dump
> 
> 
>
>   * How To: Capture a heap dump
> 
> 
>
>   * How To: Examine a Stacktrace
> 
> 
>
>   * How To: Configure Tomcat for debugging
> 
> 
>
>   * FAQ: Developing
> 
> 
>
>   * FAQ: Memory
> 
> 
>
>   * Tomcat Memory Leak Protection
> 
> 
>
>   * Sun Technical Article: Monitoring and Managing Java SE 6
> Platform Applications
> 
>
>   * Notes on using JMX clients
> 
> 
>
>
> ...
>
>   * jinfo - Prints JVM process info
> 
> 
>
>   * jstack - Prints thread stack traces
> 
> 
>
>   * jmap - Dumps heap and shows heap status
> 
> 
>
>   * jhat - Heap Analyzer Tool
> 
> 
>
>   * jcmd - Multitool intended to replace the above JDK tools
> 
> 
>
>
>
>   Profilers & Heap Analyzers
>
>   * Eclipse Memory Analyzer (MAT) 
>   * YourKit Profiler 
>  *
>
> VisualVM Docs
> 
> 
>  
>
>
> Anchor
>
>   usingjmxclients
>
>   usingjmxclients
>
> ...
>
>  1. Look into Tomcat access log
> 
> 
> (the log file generated by AccessLogValve
> 
> ).
>
>   * If your request is not listed there, then it has not been
> processed by Tomcat. You need to look elsewhere (e.g. at
> your firewall).
>

[CONF] Apache Tomcat > Troubleshooting and Diagnostics

[Felix Schumacher (Confluence)]

Title: Message Title



 
 
 
There's 2 new edits on this page 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Troubleshooting and Diagnostics 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
Felix Schumacher edited this page 
 
 
  
 
 

 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
Here's the version comments 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
Felix Schumacher edited at 10:25 AM 
 
 
  
 
 

 
 
 
 
 
 
 
 
 Correct Typo  
 
 
  
 
 
  
 
 

 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
Felix Schumacher edited at 10:24 AM 
 
 
  
 
 

 
 
 
 
 
 
 
 
 Updated Links to FAQs for Memory and Developing  
 
 
  
 
 
  
 
 

 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Here's what changed: 
 
 
 
 
 
 
 
 
 
 
 ... 
 
 How To: Capture a thread dump  
 How To: Capture a heap dump  
 How To: Examine a Stacktrace  
 How To: Configure Tomcat for debugging  
 FAQ: Developing  
 FAQ: Memory  
 Tomcat Memory Leak Protection  
 Sun Technical Article: Monitoring and Managing Java SE 6 Platform Applications  
 Notes on using JMX clients  
 ... 
 
Look into Tomcat access log (the log file generated by AccessLogValve).  
 
If your request is not listed there, then it has not been processed by Tomcat. You need to look elsewhere (e.g. at your firewall). 
You will see what IP address your client is using, and whether it is using an IPv4 (127.0.0.1) or IPv6 address (0:0:0:0:0:0:0:1). Modern operating systems can use IPv6 addresses for localhost / local network access, while external network is still using IPv4. 2. Take a thread dump. This way you will find out what Tomcat is actually doing. 
If you are troubleshooting some process that takes noticeable time, take several (three) thread dumps with some interval between them. This way you will see if there are any changes, any progress. 3. Try debugging. 
A good place for a breakpoint is org.apache.catalina.connector.CoyoteAdapter.service() method. That is the entry point from Tomcat connectors and into the Servlet engine. At that place your request has already been received and its processing starts. 
  
 ... You can also search the archives of the Tomcat users' mailing lists for previous discussions mentioning the RECYCLE_FACADES flag. 2. Read about Java ImageIO issue. Accessing response objects after their lifetime can lead to security issues in your application, such as sending responses to wrong clients, mixing up responses. If you can reproduce the issue and the above diagnostic does not show your own bug, but a bug in Apache Tomcat, ...  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Go to page history 
 
 
  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
View page 
 
 
  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
Stop watching space
• 
 
 
 
 
 
 
Manage notifications 
 
 
 
 
 
 
 
 
 
 
  
 
 
This message was sent by Atlassian Confluence 6.15.2  
 
 
  
 
 
 
 
 
 
 
 
 

Re: [CONF] Apache Tomcat > Troubleshooting and Diagnostics

[Eugène Adell]

Hello,

if I may suggest something, over the years I have found very useful jstat
and GCViewer for detecting GC suspicious behaviors. Both are of course
free, the first one giving information on a live JVM, and the second being
more interesting for an offline analysis (though it can be updated
automatically). With that you can see an OOM coming, and you also can have
some clues for the heap settings if you want to tune them. You also have a
visual proof when you perform a major upgrade and want to see if it had an
impact on memory, by comparing 2 or 3 pictures before with 2 or 3 pictures
after. They were not very often mentionned in the users list, maybe they
are underrated ?

best regards
E.A.

Le mar. 4 juin 2019 à 21:00, Felix Schumacher (Confluence) <
no-re...@apache.org> a écrit :

> There's *1 new edit* on this page
>
> [image: page icon]
> 
>  Troubleshooting
> and Diagnostics
> 
> Felix Schumacher edited this page
> Here's what changed:
>
> ...
>
>- How To: Capture a thread dump
>
> 
>- How To: Capture a heap dump
>
> 
>- How To: Examine a Stacktrace
>
> 
>- How To: Configure Tomcat for debugging
>
> 
>- FAQ: Developing
>
> 
>- FAQ: Memory
>
> 
>- Tomcat Memory Leak Protection
>
> 
>- Sun Technical Article: Monitoring and Managing Java SE 6 Platform
>Applications
>
>- Notes on using JMX clients
>
> 
>
> ...
>
>- jinfo - Prints JVM process info
>
>- jstack - Prints thread stack traces
>
> 
>- jmap - Dumps heap and shows heap status
>
>- jhat - Heap Analyzer Tool
>
>- jcmd - Multitool intended to replace the above JDK tools
>
>
> Profilers & Heap Analyzers
>
>- Eclipse Memory Analyzer (MAT) 
>- YourKit Profiler 
>-
>
>VisualVM Docs
>
> 
>
>
> Anchor
> usingjmxclients
> usingjmxclients
>
> ...
>
>1. Look into Tomcat access log
>
> 
>(the log file generated by AccessLogValve
>
> ).
>
>- If your request is not listed there, then it has not been processed
>   by Tomcat. You need to look elsewhere (e.g. at your firewall).
>   - You will see what IP address your client is using, and whether it
>   is using an IPv4 (127.0.0.1) or IPv6 address (0:0:0:0:0:0:0:1).
>   Modern operating systems can use IPv6 addresses for localhost / local
>   network access, while external network is still using IPv4.
>   2. Take a thread dump
>   
> .
>   This way you will find out what Tomcat is actually doing.
>   - If you are troubleshooting some process that takes noticeable
>   time, take several (three) thread dumps with some interval between them.
>   This way you will see if there are any changes, any progress.
>   3. Try debugging
>   
> <

[CONF] Apache Tomcat > Troubleshooting and Diagnostics

[Felix Schumacher (Confluence)]

Title: Message Title



 
 
 
There's 1 new edit on this page 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Troubleshooting and Diagnostics 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
Felix Schumacher edited this page 
 
 
  
 
 

 
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Here's what changed: 
 
 
 
 
 
 
 
 
 
 
 ... 
 
 How To: Capture a thread dump  
 How To: Capture a heap dump  
 How To: Examine a Stacktrace  
 How To: Configure Tomcat for debugging  
 FAQ: Developing  
 FAQ: Memory  
 Tomcat Memory Leak Protection  
 Sun Technical Article: Monitoring and Managing Java SE 6 Platform Applications  
 Notes on using JMX clients  
 ... 
 
 jinfo - Prints JVM process info  
 jstack - Prints thread stack traces  
 jmap - Dumps heap and shows heap status  
 jhat - Heap Analyzer Tool  
 jcmd - Multitool intended to replace the above JDK tools  
 Profilers & Heap Analyzers 
 
 Eclipse Memory Analyzer (MAT)  
 YourKit Profiler  
  VisualVM Docs    
  
 
 
 
 Anchor 
 
 
 
 
 
 
 
 
 
usingjmxclients 
 
 
 
usingjmxclients 
 
 
  
 
 
  ... 
 
Look into Tomcat access log (the log file generated by AccessLogValve).  
 
If your request is not listed there, then it has not been processed by Tomcat. You need to look elsewhere (e.g. at your firewall). 
You will see what IP address your client is using, and whether it is using an IPv4 (127.0.0.1) or IPv6 address (0:0:0:0:0:0:0:1). Modern operating systems can use IPv6 addresses for localhost / local network access, while external network is still using IPv4. 2. Take a thread dump. This way you will find out what Tomcat is actually doing. 
If you are troubleshooting some process that takes noticeable time, take several (three) thread dumps with some interval between them. This way you will see if there are any changes, any progress. 3. Try debugging. 
A good place for a breakpoint is org.apache.catalina.connector.CoyoteAdapter.service() method. That is the entry point from Tomcat connectors and into the Servlet engine. At that place your request has already been received and its processing starts. 
  
 ... The main suspect is your own web application keeping a reference to Request / Response objects outside of their life cycle.  The lifetime of the Response object is documented in the Servlet specification. Quoting from section "5.8 Lifetime of the Response Object" of Servlet 4.0 specification:   "Each response object is valid only within the scope of a servlet’s service method, or within the scope of a filter’s doFilter method, unless the associated request object has asynchronous processing enabled for the component. If asynchronous processing on the associated request is started, then the response object remains valid until complete method on AsyncContext is called."   In case of asynchronous processing, when an error occurs Tomcat notifies all registered AsyncListener}}s and then calls {{complete() automatically if none of the listeners have called it yet. (Reference: 61768)   Also see sections "2.3.3.4 Thread Safety" and "3.13 Lifetime of the Request Object" of the same specification. To troubleshoot the issue: ... You can also search the archives of the Tomcat users' mailing lists for previous discussions mentioning the RECYCLE_FACADES flag. 2. Read about Java ImageIO issue. Accessing response objects after their lifetime can lead to security issues in your application, such as sending responses to wrong clients, mixing up responses. If you can reproduce the issue and the above diagnostic does not show your own bug, but a bug in Apache Tomcat, ...  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Go to page history 
 
 
  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
View page 
 
 
  
 
 
  
 
 
  
 
 
  
 
 
 
 
 
 
 
 
 
 
Stop watching space
• 
 
 
 
 
 
 
Manage notifications 
 
 
 
 
 
 
 
 
 
 
  
 
 
This message was sent by Atlassian Confluence 6.15.2