Re: Stuck threads reading socket

[Woonsan Ko]

Thank you so much, Mark!
It helped a lot! Appreciate much for sharing the slides, too.

Cheers,

Woonsan


On Fri, Mar 4, 2016 at 3:28 AM, Mark Thomas  wrote:
> On 04/03/2016 00:51, Woonsan Ko wrote:
>> I have tomcat instances (6.0.44) which are proxied by Apache 2.4.18
>> (+mod_proxy). SSL enabled on Apache side.
>> One day it was not responding and observed that all the catalina-exec
>> threads (25 maxthreads at the moment) were stuck in reading socket in
>> the thread dumps like the following (increasing maxthreads to 128
>> circumvented the issue for now, btw):
>
> 
>
>> What would you recommend to find the root cause in this case?
>
> That looks like a common configuration error.
>
> Take a look at this:
> http://home.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
>
> Particularly slides 29 to 34.
>
> Mark
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Stuck threads reading socket

[Mark Thomas]

On 04/03/2016 00:51, Woonsan Ko wrote:
> I have tomcat instances (6.0.44) which are proxied by Apache 2.4.18
> (+mod_proxy). SSL enabled on Apache side.
> One day it was not responding and observed that all the catalina-exec
> threads (25 maxthreads at the moment) were stuck in reading socket in
> the thread dumps like the following (increasing maxthreads to 128
> circumvented the issue for now, btw):



> What would you recommend to find the root cause in this case?

That looks like a common configuration error.

Take a look at this:
http://home.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf

Particularly slides 29 to 34.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Stuck threads reading socket

[Woonsan Ko]

Hi,

I have tomcat instances (6.0.44) which are proxied by Apache 2.4.18
(+mod_proxy). SSL enabled on Apache side.
One day it was not responding and observed that all the catalina-exec
threads (25 maxthreads at the moment) were stuck in reading socket in
the thread dumps like the following (increasing maxthreads to 128
circumvented the issue for now, btw):

"catalina-exec-26537" daemon prio=10 tid=0x7f3787ff6000 nid=0x51bc
runnable [0x7f3776a4a000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at 
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:733)
at 
org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:364)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:814)
at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620)
at 
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:396)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

I want to find a way to determine the root cause(s). Maybe some bad
requests are coming in or something else.. No DDoS was reported.
So, I thought about turning on tomcat AccessLogValve with %I in format
(to print out thread name) to associate the request info with the
stuck thread. I can also get more info from apache2 access logs based
on that. However, AccessLogValve probably leaves a log only after
completing the other valves, so I'm doubtful if it can help in my
case.
What would you recommend to find the root cause in this case?

Thanks in advance,

Woonsan

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org