On Mon, Jan 2, 2012 at 3:09 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Mike,
>
> On 12/29/11 9:01 PM, Mike Wertheim wrote:
>> I did a stack dump on 7.0.23 when it was in its messed up state.
>> No threads are blocked on critical sections of my app.
>
> That's good.
>
>> For the port 8080 connector, there are 196 threads in "waiting on
>> condition" and 4 threads in "runnable".
>
> What were those 4 RUNNABLE threads doing?
They were processing POST requests from the client with a somewhat
large data payload.
Here's the stack from one of them:
"http-apr-8080-exec-169" daemon prio=10 tid=0x00002aaabc893000
nid=0x5253 runnable [0x00000000451a9000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.recvbb(Native Method)
at
org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:614)
at
org.apache.coyote.http11.InternalAprInputBuffer$SocketInputBuffer.doRead(InternalAprInputBuffer.java:656)
at
org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118)
at
org.apache.coyote.http11.InternalAprInputBuffer.doRead(InternalAprInputBuffer.java:548)
at org.apache.coyote.Request.doRead(Request.java:422)
at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:429)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315)
at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200)
at org.apache.catalina.connector.Request.readPostBody(Request.java:3096)
at
org.apache.catalina.connector.Request.parseParameters(Request.java:3049)
at org.apache.catalina.connector.Request.getParameter(Request.java:1139)
at
org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:382)
at com.mycompany.UploadServlet.service(UploadServlet.java:29)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:207)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at
org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1773)
- locked <0x00000006680fdee8> (a
org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
>
>> For the port 8443 connector, there are 81 threads in "waiting on
>> condition" and 119 threads in "runnable".
>
> Well, that looks like it might be a problem. What about the stack
> traces for THOSE threads?
They mostly look like this:
"http-apr-8443-exec-200" daemon prio=10 tid=0x00002aaac8132800
nid=0x520f runnable [0x0000000044fba000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.SSLSocket.handshake(Native Method)
at
org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(AprEndpoint.java:751)
at
org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1766)
- locked <0x000000067f3101e0> (a
org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
> When you say "slowly grinds to a halt" do you mean that there's point
> where you can no longer accept HTTP requests? How about HTTPS? When
> you get stuck, what does 'top' show? Do the threads look idle or are
> you burning up your CPU?
It definitely stops accepting some (but not all) http requests. To
answer the other questions, I'd have to run the test again (and it
takes a day to fail).
>> I don't know how to interpret this data, since I'm not really clear
>> on what the difference is between "runnable" and "waiting on
>> condition".
>
> "Waiting on condition" means that the thread is asleep, waiting for a
> notify() signal from another thread telling it to re-attempt a monitor
> lock (i.e. enter a synchronized block). "Runnable" means that the
> thread is actually doing something.
>
>> Do these numbers seem normal, or are they perhaps giving a clue as
>> to what the underlying problem might be?
>
> If those are production numbers, then you don't look like you have 1%
> HTTPS traffic -- at least not at the moment you took your thread
> dump(s). Otherwise, your connector is getting fouled-up. There's
> another thread this week about weirdness in the NIO connector that
> sounds somewhat similar, but I doubt the causes are connected. You
> might want to read that thread just the kicks.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk8CORkACgkQ9CaO5/Lv0PCAUACeJSL6ueAhjdOmiwY0qEF+oLw7
> xTUAn2nMUUwofRRlJSAX8EYMXw51W2QX
> =v+SV
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
