Re: help analyzing thread dump for Tomcat 7.0.23 issue
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mike, On 1/2/12 9:55 PM, Mike Wertheim wrote: On Mon, Jan 2, 2012 at 3:09 PM, Christopher Schultz What were those 4 RUNNABLE threads doing? They were processing POST requests from the client with a somewhat large data payload. Here's the stack from one of them: http-apr-8080-exec-169 daemon prio=10 tid=0x2aaabc893000 nid=0x5253 runnable [0x451a9000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.jni.Socket.recvbb(Native Method) at org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:614) at org.apache.coyote.http11.InternalAprInputBuffer$SocketInputBuffer.doRead(InternalAprInputBuffer.java:656) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118) at org.apache.coyote.http11.InternalAprInputBuffer.doRead(InternalAprInputBuffer.java:548) at org.apache.coyote.Request.doRead(Request.java:422) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:429) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200) at org.apache.catalina.connector.Request.readPostBody(Request.java:3096) at org.apache.catalina.connector.Request.parseParameters(Request.java:3049) at org.apache.catalina.connector.Request.getParameter(Request.java:1139) at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:382) at com.mycompany.UploadServlet.service(UploadServlet.java:29) Okay, that looks like a fairly-standard reading-the-POST-body operation. I don't think there's anything concerning in there. For the port 8443 connector, there are 81 threads in waiting on condition and 119 threads in runnable. Well, that looks like it might be a problem. What about the stack traces for THOSE threads? They mostly look like this: http-apr-8443-exec-200 daemon prio=10 tid=0x2aaac8132800 nid=0x520f runnable [0x44fba000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.jni.SSLSocket.handshake(Native Method) at org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(AprEndpoint.java:751) at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1766) - locked 0x00067f3101e0 (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Uh, oh. If you have 81 threads stuck in that state, then something is probably broken with the SSL handshake. Do you have any suspicion of some kind of DOS attack or something? I have to imagine that the APR SSL code is fairly robust. I'd be interested to see just how runnable that thread is, given that Chuck points out that a Java RUNNABLE thread might mean it's actually blocked in native code somewhere. At least blocked threads don't burn CPU time, even though they might not be accomplishing anything useful. When you say slowly grinds to a halt do you mean that there's point where you can no longer accept HTTP requests? How about HTTPS? When you get stuck, what does 'top' show? Do the threads look idle or are you burning up your CPU? It definitely stops accepting some (but not all) http requests. To answer the other questions, I'd have to run the test again (and it takes a day to fail). That's odd that the HTTP responses stall when it looks like the HTTPS connector is the one suffering. That might point to thread starvation if you are experiencing spinning threads instead of blocked ones. But in your followup you said CPU was idle. Hmm. Can you reproduce this in a test environment? If so, lower the number of threads in each connector in order to trigger the problem sooner. So, instead of 200 threads per connector, use 20 and see if you can lock it up in less time. I suspect you can. If this is in production, I highly recommend that you find yourself a test box that you can abuse while investigating this issue. Any errors shown in any logs? Look at logs/*, not just catalina.out. Also, check syslog just in case there's something in there for whatever reason. Also, please follow the 7.0.22 + fd leak with APR/native thread to see if any of that matches your observations. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8E3RsACgkQ9CaO5/Lv0PCT6wCgncxtU2TDkvynbxm2ukgsV3BL L44AnRhdSpqvMLtdvgKn7trmhcqUp0Kn =obOA -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail:
Re: help analyzing thread dump for Tomcat 7.0.23 issue
What were those 4 RUNNABLE threads doing? They were processing POST requests from the client with a somewhat large data payload. Just wanted to pop in and say that my app does also get large POST requests AND uses APR AND has problems on 7.0.23. Perhaps my FD leak is related to this? /Janne - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: help analyzing thread dump for Tomcat 7.0.23 issue
On Mon, Jan 2, 2012 at 6:55 PM, Mike Wertheim m...@hyperreal.org wrote: On Mon, Jan 2, 2012 at 3:09 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mike, On 12/29/11 9:01 PM, Mike Wertheim wrote: I did a stack dump on 7.0.23 when it was in its messed up state. No threads are blocked on critical sections of my app. That's good. For the port 8080 connector, there are 196 threads in waiting on condition and 4 threads in runnable. What were those 4 RUNNABLE threads doing? They were processing POST requests from the client with a somewhat large data payload. Here's the stack from one of them: http-apr-8080-exec-169 daemon prio=10 tid=0x2aaabc893000 nid=0x5253 runnable [0x451a9000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.jni.Socket.recvbb(Native Method) at org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:614) at org.apache.coyote.http11.InternalAprInputBuffer$SocketInputBuffer.doRead(InternalAprInputBuffer.java:656) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118) at org.apache.coyote.http11.InternalAprInputBuffer.doRead(InternalAprInputBuffer.java:548) at org.apache.coyote.Request.doRead(Request.java:422) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:429) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200) at org.apache.catalina.connector.Request.readPostBody(Request.java:3096) at org.apache.catalina.connector.Request.parseParameters(Request.java:3049) at org.apache.catalina.connector.Request.getParameter(Request.java:1139) at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:382) at com.mycompany.UploadServlet.service(UploadServlet.java:29) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:207) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539) at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1773) - locked 0x0006680fdee8 (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) For the port 8443 connector, there are 81 threads in waiting on condition and 119 threads in runnable. Well, that looks like it might be a problem. What about the stack traces for THOSE threads? They mostly look like this: http-apr-8443-exec-200 daemon prio=10 tid=0x2aaac8132800 nid=0x520f runnable [0x44fba000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.jni.SSLSocket.handshake(Native Method) at org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(AprEndpoint.java:751) at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1766) - locked 0x00067f3101e0 (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) When you say slowly
Re: help analyzing thread dump for Tomcat 7.0.23 issue
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mike, On 12/29/11 9:01 PM, Mike Wertheim wrote: I did a stack dump on 7.0.23 when it was in its messed up state. No threads are blocked on critical sections of my app. That's good. For the port 8080 connector, there are 196 threads in waiting on condition and 4 threads in runnable. What were those 4 RUNNABLE threads doing? For the port 8443 connector, there are 81 threads in waiting on condition and 119 threads in runnable. Well, that looks like it might be a problem. What about the stack traces for THOSE threads? When you say slowly grinds to a halt do you mean that there's point where you can no longer accept HTTP requests? How about HTTPS? When you get stuck, what does 'top' show? Do the threads look idle or are you burning up your CPU? I don't know how to interpret this data, since I'm not really clear on what the difference is between runnable and waiting on condition. Waiting on condition means that the thread is asleep, waiting for a notify() signal from another thread telling it to re-attempt a monitor lock (i.e. enter a synchronized block). Runnable means that the thread is actually doing something. Do these numbers seem normal, or are they perhaps giving a clue as to what the underlying problem might be? If those are production numbers, then you don't look like you have 1% HTTPS traffic -- at least not at the moment you took your thread dump(s). Otherwise, your connector is getting fouled-up. There's another thread this week about weirdness in the NIO connector that sounds somewhat similar, but I doubt the causes are connected. You might want to read that thread just the kicks. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8CORkACgkQ9CaO5/Lv0PCAUACeJSL6ueAhjdOmiwY0qEF+oLw7 xTUAn2nMUUwofRRlJSAX8EYMXw51W2QX =v+SV -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: help analyzing thread dump for Tomcat 7.0.23 issue
From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: help analyzing thread dump for Tomcat 7.0.23 issue Runnable means that the thread is actually doing something. Or that the thread is out in native code - and possibly stuck there. The OP is running APR... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: help analyzing thread dump for Tomcat 7.0.23 issue
On Mon, Jan 2, 2012 at 3:09 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mike, On 12/29/11 9:01 PM, Mike Wertheim wrote: I did a stack dump on 7.0.23 when it was in its messed up state. No threads are blocked on critical sections of my app. That's good. For the port 8080 connector, there are 196 threads in waiting on condition and 4 threads in runnable. What were those 4 RUNNABLE threads doing? They were processing POST requests from the client with a somewhat large data payload. Here's the stack from one of them: http-apr-8080-exec-169 daemon prio=10 tid=0x2aaabc893000 nid=0x5253 runnable [0x451a9000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.jni.Socket.recvbb(Native Method) at org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:614) at org.apache.coyote.http11.InternalAprInputBuffer$SocketInputBuffer.doRead(InternalAprInputBuffer.java:656) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118) at org.apache.coyote.http11.InternalAprInputBuffer.doRead(InternalAprInputBuffer.java:548) at org.apache.coyote.Request.doRead(Request.java:422) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:429) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200) at org.apache.catalina.connector.Request.readPostBody(Request.java:3096) at org.apache.catalina.connector.Request.parseParameters(Request.java:3049) at org.apache.catalina.connector.Request.getParameter(Request.java:1139) at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:382) at com.mycompany.UploadServlet.service(UploadServlet.java:29) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:207) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539) at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1773) - locked 0x0006680fdee8 (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) For the port 8443 connector, there are 81 threads in waiting on condition and 119 threads in runnable. Well, that looks like it might be a problem. What about the stack traces for THOSE threads? They mostly look like this: http-apr-8443-exec-200 daemon prio=10 tid=0x2aaac8132800 nid=0x520f runnable [0x44fba000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.jni.SSLSocket.handshake(Native Method) at org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(AprEndpoint.java:751) at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1766) - locked 0x00067f3101e0 (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) When you say slowly grinds to a halt do you mean that there's point where you can no longer accept HTTP requests? How about HTTPS? When
Re: help analyzing thread dump for Tomcat 7.0.23 issue
On 30/12/2011 02:01, Mike Wertheim wrote: I have an app that runs on Tomcat 7.0.21 and APR 1.4.7 just fine. The app serves both SSL and non-SSL traffic. SSL traffic makes up roughly 1% of the overall traffic. The Connectors for SSL and non-SSL both use the default value (200) for maxThreads. When I try to run the app on Tomcat 7.0.23 (with the same server.xml, context.xml and web.xml) and APR 1.4.7, the app slowly grinds to a halt after running on a live production server for about a day. I took heap dumps of the app on both 7.0.21 and 7.0.23 and don't see any significant difference in memory usage. Memory seems to be fine. I did a stack dump on 7.0.23 when it was in its messed up state. No threads are blocked on critical sections of my app. For the port 8080 connector, there are 196 threads in waiting on condition and 4 threads in runnable. For the port 8443 connector, there are 81 threads in waiting on condition and 119 threads in runnable. I don't know how to interpret this data, since I'm not really clear on what the difference is between runnable and waiting on condition. Do these numbers seem normal, or are they perhaps giving a clue as to what the underlying problem might be? Which Connector are you using? Can you please remove the XML comments and obscure username/password/ip addresses if applicable in your server.xml and then post it inline here? p -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: help analyzing thread dump for Tomcat 7.0.23 issue
I'm using the APR Connector. Here's my server.xml: ?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=on / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / Resource name=mail/Session auth=Container type=javax.mail.Session mail.smtp.host=localhost/ /GlobalNamingResources Service name=Catalina Connector port=8080 protocol=HTTP/1.1 URIEncoding=utf-8 maxKeepAliveRequests=3 keepAliveTimeout=3000 redirectPort=8443 / Connector port=8443 URIEncoding=utf-8 maxKeepAliveRequests=3 keepAliveTimeout=3000 scheme=https secure=true SSLEnabled=true SSLCertificateFile=/home/user/ssl/2010.www.surfcanyon.com.crt SSLCertificateKeyFile=/home/user/ssl/www.surfcanyon.com.key SSLCertificateChainFile=/home/user/ssl/intermediateCA.cer clientAuth=false sslProtocol=TLS/ Engine name=Catalina defaultHost=localhost Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase/ Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true Valve className=org.apache.catalina.valves.AccessLogValve directory=logs prefix=localhost_access_log. suffix=.txt pattern=%I %t %U %T resolveHosts=false/ /Host /Engine /Service /Server On Sat, Dec 31, 2011 at 5:39 AM, Pid p...@pidster.com wrote: On 30/12/2011 02:01, Mike Wertheim wrote: I have an app that runs on Tomcat 7.0.21 and APR 1.4.7 just fine. The app serves both SSL and non-SSL traffic. SSL traffic makes up roughly 1% of the overall traffic. The Connectors for SSL and non-SSL both use the default value (200) for maxThreads. When I try to run the app on Tomcat 7.0.23 (with the same server.xml, context.xml and web.xml) and APR 1.4.7, the app slowly grinds to a halt after running on a live production server for about a day. I took heap dumps of the app on both 7.0.21 and 7.0.23 and don't see any significant difference in memory usage. Memory seems to be fine. I did a stack dump on 7.0.23 when it was in its messed up state. No threads are blocked on critical sections of my app. For the port 8080 connector, there are 196 threads in waiting on condition and 4 threads in runnable. For the port 8443 connector, there are 81 threads in waiting on condition and 119 threads in runnable. I don't know how to interpret this data, since I'm not really clear on what the difference is between runnable and waiting on condition. Do these numbers seem normal, or are they perhaps giving a clue as to what the underlying problem might be? Which Connector are you using? Can you please remove the XML comments and obscure username/password/ip addresses if applicable in your server.xml and then post it inline here? p -- [key:62590808] - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: help analyzing thread dump for Tomcat 7.0.23 issue
On 31 Dec 2011, at 16:14, Mike Wertheim m...@hyperreal.org wrote: I'm using the APR Connector. Here's my server.xml: ?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=on / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / No memory leak prevention listeners? GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / Resource name=mail/Session auth=Container type=javax.mail.Session mail.smtp.host=localhost/ /GlobalNamingResources Service name=Catalina Connector port=8080 protocol=HTTP/1.1 URIEncoding=utf-8 maxKeepAliveRequests=3 keepAliveTimeout=3000 redirectPort=8443 / Connector port=8443 URIEncoding=utf-8 maxKeepAliveRequests=3 keepAliveTimeout=3000 scheme=https secure=true SSLEnabled=true SSLCertificateFile=/home/user/ssl/2010.www.surfcanyon.com.crt SSLCertificateKeyFile=/home/user/ssl/www.surfcanyon.com.key SSLCertificateChainFile=/home/user/ssl/intermediateCA.cer clientAuth=false sslProtocol=TLS/ Can you configure an Executor? p Engine name=Catalina defaultHost=localhost Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase/ Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true Valve className=org.apache.catalina.valves.AccessLogValve directory=logs prefix=localhost_access_log. suffix=.txt pattern=%I %t %U %T resolveHosts=false/ /Host /Engine /Service /Server On Sat, Dec 31, 2011 at 5:39 AM, Pid p...@pidster.com wrote: On 30/12/2011 02:01, Mike Wertheim wrote: I have an app that runs on Tomcat 7.0.21 and APR 1.4.7 just fine. The app serves both SSL and non-SSL traffic. SSL traffic makes up roughly 1% of the overall traffic. The Connectors for SSL and non-SSL both use the default value (200) for maxThreads. When I try to run the app on Tomcat 7.0.23 (with the same server.xml, context.xml and web.xml) and APR 1.4.7, the app slowly grinds to a halt after running on a live production server for about a day. I took heap dumps of the app on both 7.0.21 and 7.0.23 and don't see any significant difference in memory usage. Memory seems to be fine. I did a stack dump on 7.0.23 when it was in its messed up state. No threads are blocked on critical sections of my app. For the port 8080 connector, there are 196 threads in waiting on condition and 4 threads in runnable. For the port 8443 connector, there are 81 threads in waiting on condition and 119 threads in runnable. I don't know how to interpret this data, since I'm not really clear on what the difference is between runnable and waiting on condition. Do these numbers seem normal, or are they perhaps giving a clue as to what the underlying problem might be? Which Connector are you using? Can you please remove the XML comments and obscure username/password/ip addresses if applicable in your server.xml and then post it inline here? p -- [key:62590808] - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: help analyzing thread dump for Tomcat 7.0.23 issue
On Sat, Dec 31, 2011 at 12:53 PM, Pid * p...@pidster.com wrote: On 31 Dec 2011, at 16:14, Mike Wertheim m...@hyperreal.org wrote: I'm using the APR Connector. Here's my server.xml: ?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=on / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / No memory leak prevention listeners? Nope. I'm a luddite. I kill the Tomcat UNIX process and restart it every time I re-deploy the app. GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / Resource name=mail/Session auth=Container type=javax.mail.Session mail.smtp.host=localhost/ /GlobalNamingResources Service name=Catalina Connector port=8080 protocol=HTTP/1.1 URIEncoding=utf-8 maxKeepAliveRequests=3 keepAliveTimeout=3000 redirectPort=8443 / Connector port=8443 URIEncoding=utf-8 maxKeepAliveRequests=3 keepAliveTimeout=3000 scheme=https secure=true SSLEnabled=true SSLCertificateFile=/home/user/ssl/2010.www.surfcanyon.com.crt SSLCertificateKeyFile=/home/user/ssl/www.surfcanyon.com.key SSLCertificateChainFile=/home/user/ssl/intermediateCA.cer clientAuth=false sslProtocol=TLS/ Can you configure an Executor? I tried that a while back and it didn't make a difference. p Engine name=Catalina defaultHost=localhost Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase/ Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true Valve className=org.apache.catalina.valves.AccessLogValve directory=logs prefix=localhost_access_log. suffix=.txt pattern=%I %t %U %T resolveHosts=false/ /Host /Engine /Service /Server On Sat, Dec 31, 2011 at 5:39 AM, Pid p...@pidster.com wrote: On 30/12/2011 02:01, Mike Wertheim wrote: I have an app that runs on Tomcat 7.0.21 and APR 1.4.7 just fine. The app serves both SSL and non-SSL traffic. SSL traffic makes up roughly 1% of the overall traffic. The Connectors for SSL and non-SSL both use the default value (200) for maxThreads. When I try to run the app on Tomcat 7.0.23 (with the same server.xml, context.xml and web.xml) and APR 1.4.7, the app slowly grinds to a halt after running on a live production server for about a day. I took heap dumps of the app on both 7.0.21 and 7.0.23 and don't see any significant difference in memory usage. Memory seems to be fine. I did a stack dump on 7.0.23 when it was in its messed up state. No threads are blocked on critical sections of my app. For the port 8080 connector, there are 196 threads in waiting on condition and 4 threads in runnable. For the port 8443 connector, there are 81 threads in waiting on condition and 119 threads in runnable. I don't know how to interpret this data, since I'm not really clear on what the difference is between runnable and waiting on condition. Do these numbers seem normal, or are they perhaps giving a clue as to what the underlying problem might be? Which Connector are you using? Can you please remove the XML comments and obscure username/password/ip addresses if applicable in your server.xml and then post it inline here? p -- [key:62590808] - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
help analyzing thread dump for Tomcat 7.0.23 issue
I have an app that runs on Tomcat 7.0.21 and APR 1.4.7 just fine. The app serves both SSL and non-SSL traffic. SSL traffic makes up roughly 1% of the overall traffic. The Connectors for SSL and non-SSL both use the default value (200) for maxThreads. When I try to run the app on Tomcat 7.0.23 (with the same server.xml, context.xml and web.xml) and APR 1.4.7, the app slowly grinds to a halt after running on a live production server for about a day. I took heap dumps of the app on both 7.0.21 and 7.0.23 and don't see any significant difference in memory usage. Memory seems to be fine. I did a stack dump on 7.0.23 when it was in its messed up state. No threads are blocked on critical sections of my app. For the port 8080 connector, there are 196 threads in waiting on condition and 4 threads in runnable. For the port 8443 connector, there are 81 threads in waiting on condition and 119 threads in runnable. I don't know how to interpret this data, since I'm not really clear on what the difference is between runnable and waiting on condition. Do these numbers seem normal, or are they perhaps giving a clue as to what the underlying problem might be? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
