Re: Socket Read long running
On 1/21/16 4:13 AM, Christopher Schultz wrote: Rallavagu, On 1/19/16 6:14 PM, Rallavagu wrote: On 1/19/16 2:43 PM, Mark Thomas wrote: On 19/01/2016 22:36, Rallavagu wrote: Also, it could be keep-alive for client connection as well. In any case, how long a keep-alive connection will be in this state by default? Thanks. This behaviour is entirely normal. Why are you concerned about it? I was analyzing thread dump as the application experiences sudden high response times and eventually becomes normal. Regarding how long the thread will be in this state, the default keep-alive timeout for the HTTP BIO connector can be found in the documentation. (Yes, I do happen to know what it is but think of this as an exercise for the reader.) From the documentation keepAliveTimeout defaults to connectionTimeout. "The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite) timeout. The default value is 6 (i.e. 60 seconds) but note that the standard server.xml that ships with Tomcat sets this to 2 (i.e. 20 seconds). Unless disableUploadTimeout is set to false, this timeout will also be used when reading the request body (if any)." You might want to consider switching to an NIO-based connector. The NIO-based connectors do not block a request-processing thread during the keep-alive wait time, so fewer threads can handle the same number of actual incoming requests (instead of waiting-around potentially doing no additional work). You may still have a response-time problem after that, but it won't be due to the threading model. If your load-balancer configured to maintain keep-alive connections to your Tomcat instance(s)? If so, what are the details of that configuration? They have configured for a pool of 50 keep-alive connections with 10 seconds timeout. Thanks -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Socket Read long running
Rallavagu, On 1/19/16 6:14 PM, Rallavagu wrote: > > > On 1/19/16 2:43 PM, Mark Thomas wrote: >> On 19/01/2016 22:36, Rallavagu wrote: >>> Also, it could be keep-alive for client connection as well. In any case, >>> how long a keep-alive connection will be in this state by default? >>> Thanks. >> >> This behaviour is entirely normal. Why are you concerned about it? > > I was analyzing thread dump as the application experiences sudden high > response times and eventually becomes normal. > >> >> Regarding how long the thread will be in this state, the default >> keep-alive timeout for the HTTP BIO connector can be found in the >> documentation. >> (Yes, I do happen to know what it is but think of this as an exercise >> for the reader.) > > From the documentation keepAliveTimeout defaults to connectionTimeout. > > > "The number of milliseconds this Connector will wait, after accepting a > connection, for the request URI line to be presented. Use a value of -1 > to indicate no (i.e. infinite) timeout. The default value is 6 (i.e. > 60 seconds) but note that the standard server.xml that ships with Tomcat > sets this to 2 (i.e. 20 seconds). Unless disableUploadTimeout is set > to false, this timeout will also be used when reading the request body > (if any)." You might want to consider switching to an NIO-based connector. The NIO-based connectors do not block a request-processing thread during the keep-alive wait time, so fewer threads can handle the same number of actual incoming requests (instead of waiting-around potentially doing no additional work). You may still have a response-time problem after that, but it won't be due to the threading model. If your load-balancer configured to maintain keep-alive connections to your Tomcat instance(s)? If so, what are the details of that configuration? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Socket Read long running
On 19/01/2016 22:36, Rallavagu wrote: > Also, it could be keep-alive for client connection as well. In any case, > how long a keep-alive connection will be in this state by default? Thanks. This behaviour is entirely normal. Why are you concerned about it? Regarding how long the thread will be in this state, the default keep-alive timeout for the HTTP BIO connector can be found in the documentation. (Yes, I do happen to know what it is but think of this as an exercise for the reader.) Mark > > On 1/19/16 2:24 PM, Rallavagu wrote: >> Thanks Mark. It seems to be running for almost 10 seconds and there is a >> Load Balancer between. Is it a suspect? >> >> On 1/19/16 2:09 PM, Mark Thomas wrote: >>> On 19/01/2016 21:42, Rallavagu wrote: I have this long running thread. It appears to be reading but the stack trace does not give much of a clue. Could anyone help with where to start? Thanks. >>> >>> Looks like an HTTP keep-alive connection waiting for the next request. >>> >>> Mark >>> >>> Tomcat 7.0.42 with JDK 7 "tomcat-exec-2655" daemon prio=10 tid=0x7fc459061000 nid=0x6a58 runnable [0x7fc4a67e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:991) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked <0x000799d09cb8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00072eebc6c0> (a java.util.concurrent.ThreadPoolExecutor$Worker) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Socket Read long running
Also, it could be keep-alive for client connection as well. In any case, how long a keep-alive connection will be in this state by default? Thanks. On 1/19/16 2:24 PM, Rallavagu wrote: Thanks Mark. It seems to be running for almost 10 seconds and there is a Load Balancer between. Is it a suspect? On 1/19/16 2:09 PM, Mark Thomas wrote: On 19/01/2016 21:42, Rallavagu wrote: I have this long running thread. It appears to be reading but the stack trace does not give much of a clue. Could anyone help with where to start? Thanks. Looks like an HTTP keep-alive connection waiting for the next request. Mark Tomcat 7.0.42 with JDK 7 "tomcat-exec-2655" daemon prio=10 tid=0x7fc459061000 nid=0x6a58 runnable [0x7fc4a67e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:991) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked <0x000799d09cb8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00072eebc6c0> (a java.util.concurrent.ThreadPoolExecutor$Worker) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Socket Read long running
On 19/01/2016 21:42, Rallavagu wrote: > I have this long running thread. It appears to be reading but the stack > trace does not give much of a clue. Could anyone help with where to > start? Thanks. Looks like an HTTP keep-alive connection waiting for the next request. Mark > > Tomcat 7.0.42 with JDK 7 > > "tomcat-exec-2655" daemon prio=10 tid=0x7fc459061000 nid=0x6a58 > runnable [0x7fc4a67e6000] >java.lang.Thread.State: RUNNABLE > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:152) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at > org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519) > > at > org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) > > at > org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) > > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:991) > > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) > > at > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) > > - locked <0x000799d09cb8> (a > org.apache.tomcat.util.net.SocketWrapper) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > > at java.lang.Thread.run(Thread.java:745) > >Locked ownable synchronizers: > - <0x00072eebc6c0> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Socket Read long running
Thanks Mark. It seems to be running for almost 10 seconds and there is a Load Balancer between. Is it a suspect? On 1/19/16 2:09 PM, Mark Thomas wrote: On 19/01/2016 21:42, Rallavagu wrote: I have this long running thread. It appears to be reading but the stack trace does not give much of a clue. Could anyone help with where to start? Thanks. Looks like an HTTP keep-alive connection waiting for the next request. Mark Tomcat 7.0.42 with JDK 7 "tomcat-exec-2655" daemon prio=10 tid=0x7fc459061000 nid=0x6a58 runnable [0x7fc4a67e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:991) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked <0x000799d09cb8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00072eebc6c0> (a java.util.concurrent.ThreadPoolExecutor$Worker) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Socket Read long running
On 1/19/16 2:43 PM, Mark Thomas wrote: On 19/01/2016 22:36, Rallavagu wrote: Also, it could be keep-alive for client connection as well. In any case, how long a keep-alive connection will be in this state by default? Thanks. This behaviour is entirely normal. Why are you concerned about it? I was analyzing thread dump as the application experiences sudden high response times and eventually becomes normal. Regarding how long the thread will be in this state, the default keep-alive timeout for the HTTP BIO connector can be found in the documentation. (Yes, I do happen to know what it is but think of this as an exercise for the reader.) From the documentation keepAliveTimeout defaults to connectionTimeout. "The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite) timeout. The default value is 6 (i.e. 60 seconds) but note that the standard server.xml that ships with Tomcat sets this to 2 (i.e. 20 seconds). Unless disableUploadTimeout is set to false, this timeout will also be used when reading the request body (if any)." Thanks Mark On 1/19/16 2:24 PM, Rallavagu wrote: Thanks Mark. It seems to be running for almost 10 seconds and there is a Load Balancer between. Is it a suspect? On 1/19/16 2:09 PM, Mark Thomas wrote: On 19/01/2016 21:42, Rallavagu wrote: I have this long running thread. It appears to be reading but the stack trace does not give much of a clue. Could anyone help with where to start? Thanks. Looks like an HTTP keep-alive connection waiting for the next request. Mark Tomcat 7.0.42 with JDK 7 "tomcat-exec-2655" daemon prio=10 tid=0x7fc459061000 nid=0x6a58 runnable [0x7fc4a67e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:991) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked <0x000799d09cb8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00072eebc6c0> (a java.util.concurrent.ThreadPoolExecutor$Worker) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
