Re: currentThreadCount and currentThreadsBusy
On 10/31/16 12:54 PM, Mark Thomas wrote: On 31/10/2016 19:52, Rallavagu wrote: As per https://tomcat.apache.org/tomcat-7.0-doc/config/executor.html it appears "maxIdleTime" could be what I am looking for. Is there a corresponding config parameter for internal thread pool? Which part of "hard-coded" wasn't clear? Well, I was not sure if is referred to "maxIdleTimeout" or not and I am not seeing thread count to diminish hence had to ask. So, I guess the recommendation is to use Executor instead? Mark On 10/31/16 12:44 PM, Mark Thomas wrote: On 31/10/2016 19:38, Rallavagu wrote: Here is the configuration snippet. As you can see, _not_ using executor. Thanks. When using the internal thread pool, that delay is hard-coded to 60s. If you want to configure it, use an Executor. Mark On 10/31/16 11:58 AM, Christopher Schultz wrote: Rallavagu, On 10/31/16 1:59 PM, Rallavagu wrote: All, Tomcat 7.0.70 with bio connector I have been monitoring JMX for "currentThreadCount" and "currentThreadsBusy". If I understand correctly, currentThreadCount would not exceed "maxThreads" configuration and "currentThreadsBusy" apparently shows "busy" threads at that time. I am wondering if there is a "timeout" (or other config/parameter) for not busy threads (for long time) to be removed so currentThreadCount would be reduced. Thanks. Are you using an ? Perhaps you could post your and also configuration if you are using one. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: currentThreadCount and currentThreadsBusy
As per https://tomcat.apache.org/tomcat-7.0-doc/config/executor.html it appears "maxIdleTime" could be what I am looking for. Is there a corresponding config parameter for internal thread pool? On 10/31/16 12:44 PM, Mark Thomas wrote: On 31/10/2016 19:38, Rallavagu wrote: Here is the configuration snippet. As you can see, _not_ using executor. Thanks. When using the internal thread pool, that delay is hard-coded to 60s. If you want to configure it, use an Executor. Mark On 10/31/16 11:58 AM, Christopher Schultz wrote: Rallavagu, On 10/31/16 1:59 PM, Rallavagu wrote: All, Tomcat 7.0.70 with bio connector I have been monitoring JMX for "currentThreadCount" and "currentThreadsBusy". If I understand correctly, currentThreadCount would not exceed "maxThreads" configuration and "currentThreadsBusy" apparently shows "busy" threads at that time. I am wondering if there is a "timeout" (or other config/parameter) for not busy threads (for long time) to be removed so currentThreadCount would be reduced. Thanks. Are you using an ? Perhaps you could post your and also configuration if you are using one. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: currentThreadCount and currentThreadsBusy
On 10/31/16 12:44 PM, Mark Thomas wrote: On 31/10/2016 19:38, Rallavagu wrote: Here is the configuration snippet. As you can see, _not_ using executor. Thanks. When using the internal thread pool, that delay is hard-coded to 60s. If you want to configure it, use an Executor. Mark Thanks Mark. What I am noticing is that "currentThreadCount" remain same even after when there is not much of load. On 10/31/16 11:58 AM, Christopher Schultz wrote: Rallavagu, On 10/31/16 1:59 PM, Rallavagu wrote: All, Tomcat 7.0.70 with bio connector I have been monitoring JMX for "currentThreadCount" and "currentThreadsBusy". If I understand correctly, currentThreadCount would not exceed "maxThreads" configuration and "currentThreadsBusy" apparently shows "busy" threads at that time. I am wondering if there is a "timeout" (or other config/parameter) for not busy threads (for long time) to be removed so currentThreadCount would be reduced. Thanks. Are you using an ? Perhaps you could post your and also configuration if you are using one. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: currentThreadCount and currentThreadsBusy
Here is the configuration snippet. As you can see, _not_ using executor. Thanks. On 10/31/16 11:58 AM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Rallavagu, On 10/31/16 1:59 PM, Rallavagu wrote: All, Tomcat 7.0.70 with bio connector I have been monitoring JMX for "currentThreadCount" and "currentThreadsBusy". If I understand correctly, currentThreadCount would not exceed "maxThreads" configuration and "currentThreadsBusy" apparently shows "busy" threads at that time. I am wondering if there is a "timeout" (or other config/parameter) for not busy threads (for long time) to be removed so currentThreadCount would be reduced. Thanks. Are you using an ? Perhaps you could post your and also configuration if you are using one. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJYF5RsAAoJEBzwKT+lPKRYOMIP/Axsb/+n3iy/PvgM2/zibK2X aCiFjwH1bdInzCnPwaO02B6rYTgxAIEjgcBSMCTjBh7cyB9Okq1I5yo1ylVOHprd GECbu401or1FpVfj4JPLL/yK8bS2m8cZ+s+JL4SKkhYIxzeNL/bm89v+tEXmAViw 6WmvgxF7zqswQBzkmkk2Z+20JnbqsLvXhXang25/BeAPFt9Q/ojHz96N8i3ENDMf 5YiXXukGIZjtLFqtsA1V+YpuehkzYYJxkgO2viipKP1B8hOHggjh+GpV4N0AXbwv a//NcYW5AgdH38jWueEdvrTftfX+6PMc804HcdCVUkJK64zsyR3YwapyW8rZSir8 UOaELgfh/TOF0z7LK4hIRBQwZgKczj9IKJcWZsT/JYVmt3HLVyNqwo0iIw2iYbgQ 44iirJy40apjPabC7I9fJY8hRrv2JIrQg0rti2CEFH7LFGXsCaNayveKngZlPOy9 6/ZrzF2OVkzhDGV2WwCzjPak7t20OvN8F3+KLu+ldPrbWNKw15DbBJCeFUs0U02E Z5gFRvP9CiVV3qywlq4E/qtDNkMNAW/N+RBxn/KjSFosHIuZ2CwlVB993/FsDbQW d1hev6Xshr2STepoyn4++/PQ5mdrQRPOK+lMKBXtLrKRdzauAUnt3Dkgm204z4o6 vTqaaUzfscZFXPWxgdCM =X1cd -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
currentThreadCount and currentThreadsBusy
All, Tomcat 7.0.70 with bio connector I have been monitoring JMX for "currentThreadCount" and "currentThreadsBusy". If I understand correctly, currentThreadCount would not exceed "maxThreads" configuration and "currentThreadsBusy" apparently shows "busy" threads at that time. I am wondering if there is a "timeout" (or other config/parameter) for not busy threads (for long time) to be removed so currentThreadCount would be reduced. Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
X509Factory.getFromCache
Tomcat 7.0.70, JDK 1.7.0_80 $java -version java version "1.7.0_80" Java(TM) SE Runtime Environment (build 1.7.0_80-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode) There are many threads BLOCKED as below and due to this these threads are very slow. Anybody else experienced this? Am I hitting following JDK bug? https://bugs.openjdk.java.net/browse/JDK-6432000 "http-bio-28080-exec-1155" daemon prio=10 tid=0x7feca808e000 nid=0x5a31 waiting for monitor entry [0x7feba3cf8000] java.lang.Thread.State: BLOCKED (on object monitor) at sun.security.provider.X509Factory.getFromCache(X509Factory.java:215) - waiting to lock <0x0006a00be6a8> (a java.lang.Class for sun.security.provider.X509Factory) at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:94) at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:747) - locked <0x0006dd58da90> (a java.util.Hashtable) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55) at java.security.KeyStore.load(KeyStore.java:1226) at sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(TrustManagerFactoryImpl.java:221) at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:51) at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) at sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:79) at javax.net.ssl.SSLContext.init(SSLContext.java:283) at org.apache.http.ssl.SSLContexts.createDefault(SSLContexts.java:55) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.getSocketFactory(SSLConnectionSocketFactory.java:172) at org.apache.http.impl.conn.BasicHttpClientConnectionManager.getDefaultRegistry(BasicHttpClientConnectionManager.java:117) at org.apache.http.impl.conn.BasicHttpClientConnectionManager.(BasicHttpClientConnectionManager.java:161) at com.dice.api.util.service.DiceHTTPClient.getGetDeleteResponse(DiceHTTPClient.java:162) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Long running socket reads
On 10/14/16 1:01 AM, Mark Thomas wrote:
On 12/10/2016 16:22, Rallavagu wrote:
Tomcat 7.0.70 - Sun JDK 1.7.0_80
I have following long running thread (almost 5 sec).
No you don't. That thread isn't doing it any work. It is blocking,
waiting for I/O.
Well, I got in impression because the thread shows as "RUNNABLE" in tow
thread dumps in a row.
It appears to be reading data from socket
Sort of correct. It is waiting for data to arrive.
I suppose it is waiting for the data to arrive from slow clients?
(external resource potentially).
Wrong.
Wonder how I
could go about debug these kind of threads to understand which external
resource is it spending more time on reading. Thank.
The stack trace contains class names and line numbers. The source code
is freely available. While the class names themselves provide a broad
hint as to what this thread is waiting for, you always have the option
of looking at the source code.
In this case start with these:
AbstractHttp11Processor line 994
Http11Processor line 168
Here is what at line 168 of Http1Processor
if (!inputBuffer.fill()) {
throw new EOFException(sm.getString("iib.eof.error"));
}
I couldnt make much out of this. But, I assume that this thread is
waiting for client to finish sending the data?
Thanks for the response.
Mark
"http-bio-28080-exec-523" daemon prio=10 tid=0x7fc5e403e000
nid=0x3cbd runnable [0x7fc51ce0d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
at
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at
org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:994)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:623)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
- locked <0x0006e20ac9e8> (a
org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Long running socket reads
Tomcat 7.0.70 - Sun JDK 1.7.0_80 I have following long running thread (almost 5 sec). It appears to be reading data from socket (external resource potentially). Wonder how I could go about debug these kind of threads to understand which external resource is it spending more time on reading. Thank. "http-bio-28080-exec-523" daemon prio=10 tid=0x7fc5e403e000 nid=0x3cbd runnable [0x7fc51ce0d000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501) at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:994) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:623) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked <0x0006e20ac9e8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Why is Tomcat sending "Connection: close?"
One thing I would check is if it is Tomcat that is sending it or an intermediary load balancer. On 8/31/16 8:50 AM, john.e.gr...@wellsfargo.com wrote: All, I'm using Tomcat 7.0.70 and am having trouble understanding why Tomcat is sending "Connection: close" in the response header as often as it is. With almost no load on the server, I get "Connection: close" pretty much every time. The client is sending "Connection: keep-alive" but it doesn't seem to matter. HTTP protocol is 1.1 and response code is 200. In other cases I've seen Tomcat behave exactly the way the doc says the below config should behave (100 requests per connection as long as the timeout is not exceeded) but not this time. Any idea why this is occurring or where to look to debug it? I've tried setting breakpoints in AbstractHttp11Processor where "Connection: close" is set, but it's not hit. Thanks John Here is my connector config: - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat accept count tuning
Tomcat 7.0.47 running on Linux I have started investigating after noticing following messages from "dmesg" output on a production server. "possible SYN flooding on port 28080. Sending cookies." Started looking into this as the connections to this server are timing out (Connect Timeout errors). Upon further investigation, it appears to me that Linux's kernel maintain two different queues one for SYN and one for ESTABLISHED/accept connections. Both are determined by following parameters. $ cat /proc/sys/net/ipv4/tcp_max_syn_backlog 2048 $ cat /proc/sys/net/core/somaxconn 128 Also, it appears that the second parameter (accept count) is determined by the application. For tomcat it defaults to 100. As per this document - http://blog.dubbelboer.com/2012/04/09/syn-cookies.html above two parameters could be tuned to increase the accepted connections. Wondering if Tomcat's "acceptCount" (http://tomcat.apache.org/tomcat-7.0-doc/config/http.html) parameter is related to "somaxconn" for tuning. Thanks in advance for your comments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Multiple Class Loaders
Chris, Thanks for the response. As I have mentioned, there are bugs in application code that prevents WebAppClassLoader from GC collection. Wondering what happens when two instances are running one is pinned due to application bugs and one newly deployed. For instance, would the pinned instance be active? On 4/4/16 8:03 PM, Christopher Schultz wrote: Rallavagu, On 4/4/16 8:13 PM, Rallavagu wrote: Tomcat 7.0.47, JDK 7 When an app is hot deployed in-place by simply copying the .war file into webapps directory, the old webappclassloader is not cleared completely because of inefficient context shutdown from the app. In this case, two instances of WebAppClassLoader are running for the same context root (at least appears that way from heap dump). Wondering what would be the expected behavior if there is any? I understand that this is the problem with deployment and application that is written but wondering how would Tomcat behave in this case. Tomcat behaves correctly. Your web application must have code in itself or one of its dependent libraries which pins the context's ClassLoader in memory. Have a look at https://home.apache.org/~markt/presentations/2010-08-05-Memory-Leaks-JavaOne-60mins.pdf for information about ClassLoader-pinning leaks, how to find them, and how to fix them. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Multiple Class Loaders
Tomcat 7.0.47, JDK 7 When an app is hot deployed in-place by simply copying the .war file into webapps directory, the old webappclassloader is not cleared completely because of inefficient context shutdown from the app. In this case, two instances of WebAppClassLoader are running for the same context root (at least appears that way from heap dump). Wondering what would be the expected behavior if there is any? I understand that this is the problem with deployment and application that is written but wondering how would Tomcat behave in this case. Thanks - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: long running socketwrite
On 3/30/16 10:25 AM, Rallavagu wrote: On 3/30/16 9:55 AM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rallavagu, On 3/30/16 11:54 AM, Rallavagu wrote: Tomcat 7.0.47, JDK 7 I have following long running socketwrite thread (more than 10 sec). Wondering what could cause this so I can further look and investigate. "http-bio-28080-exec-1497" daemon prio=10 tid=0x7f812c230800 nid=0x72fa runnable [0x7f80010f9000] java.lang.Thread.State: RUNNABLE at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) at java.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalO utputBuffer.java:215) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuff er.java:119) at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11 Processor.java:805) at org.apache.coyote.Response.action(Response.java:174) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:3 66) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333 ) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStr eam.java:101) at org.springframework.security.web.context.SaveContextOnUpdateOrErrorRes ponseWrapper$SaveContextServletOutputStream.flush(SaveContextOnUpdateOrE rrorResponseWrapper.java:354) at com.sun.jersey.spi.container.servlet.WebComponent$Writer.flush(WebComp onent.java:308) at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream. flush(ContainerResponse.java:146) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) - locked <0x00075a610750> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at java.io.BufferedWriter.flush(BufferedWriter.java:254) - locked <0x00075a610750> (a java.io.OutputStreamWriter) at com.sun.jersey.core.util.ReaderWriter.writeToAsString(ReaderWriter.jav a:191) at com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.write ToAsString(AbstractMessageReaderWriterProvider.java:128) at com.sun.jersey.core.impl.provider.entity.StringProvider.writeTo(String Provider.java:88) at com.sun.jersey.core.impl.provider.entity.StringProvider.writeTo(String Provider.java:58) at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse .java:302) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleReque st(WebApplicationImpl.java:1510) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleReques t(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleReques t(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent .java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletC ontainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletC ontainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) This is during slow response times where many threads were opened (Tomcat configured with 500). Also, thread dump shows that 240 of tomcat threads are in following state which suggests that they are idle. "http-bio-28080-exec-1230" daemon prio=10 tid=0x7f812c361800 nid=0x29a0 waiting on condition [0x7f810a7e6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00071d193f50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject. awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java :467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.jav a:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java:615) at java.lang.Thread.run(Thread.java:744) Any clues would help. Thanks. So ~50% of your request-processing threads are active, and at least one of them is hanging on socket.write(). I would imagine you have a slow or unreliable client (e.g. mobile phone with a bad connection) or the client has disappeared but the socket hasn't closed, yet. You can set a write timeouts on your sockets, but you mig
Re: long running socketwrite
On 3/30/16 9:55 AM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rallavagu, On 3/30/16 11:54 AM, Rallavagu wrote: Tomcat 7.0.47, JDK 7 I have following long running socketwrite thread (more than 10 sec). Wondering what could cause this so I can further look and investigate. "http-bio-28080-exec-1497" daemon prio=10 tid=0x7f812c230800 nid=0x72fa runnable [0x7f80010f9000] java.lang.Thread.State: RUNNABLE at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) at java.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalO utputBuffer.java:215) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuff er.java:119) at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11 Processor.java:805) at org.apache.coyote.Response.action(Response.java:174) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:3 66) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333 ) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStr eam.java:101) at org.springframework.security.web.context.SaveContextOnUpdateOrErrorRes ponseWrapper$SaveContextServletOutputStream.flush(SaveContextOnUpdateOrE rrorResponseWrapper.java:354) at com.sun.jersey.spi.container.servlet.WebComponent$Writer.flush(WebComp onent.java:308) at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream. flush(ContainerResponse.java:146) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) - locked <0x00075a610750> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at java.io.BufferedWriter.flush(BufferedWriter.java:254) - locked <0x00075a610750> (a java.io.OutputStreamWriter) at com.sun.jersey.core.util.ReaderWriter.writeToAsString(ReaderWriter.jav a:191) at com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.write ToAsString(AbstractMessageReaderWriterProvider.java:128) at com.sun.jersey.core.impl.provider.entity.StringProvider.writeTo(String Provider.java:88) at com.sun.jersey.core.impl.provider.entity.StringProvider.writeTo(String Provider.java:58) at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse .java:302) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleReque st(WebApplicationImpl.java:1510) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleReques t(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleReques t(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent .java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletC ontainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletC ontainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) This is during slow response times where many threads were opened (Tomcat configured with 500). Also, thread dump shows that 240 of tomcat threads are in following state which suggests that they are idle. "http-bio-28080-exec-1230" daemon prio=10 tid=0x7f812c361800 nid=0x29a0 waiting on condition [0x7f810a7e6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00071d193f50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject. awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java :467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.jav a:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java:615) at java.lang.Thread.run(Thread.java:744) Any clues would help. Thanks. So ~50% of your request-processing threads are active, and at least one of them is hanging on socket.write(). I would imagine you have a slow or unreliable client (e.g. mobile phone with a bad connection) or the client has disappeared but the socket hasn't closed, yet. You can set a write timeouts on your sockets, but you might end up with more server-side errors and cran
long running socketwrite
Tomcat 7.0.47, JDK 7 I have following long running socketwrite thread (more than 10 sec). Wondering what could cause this so I can further look and investigate. "http-bio-28080-exec-1497" daemon prio=10 tid=0x7f812c230800 nid=0x72fa runnable [0x7f80010f9000] java.lang.Thread.State: RUNNABLE at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) at java.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119) at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:805) at org.apache.coyote.Response.action(Response.java:174) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101) at org.springframework.security.web.context.SaveContextOnUpdateOrErrorResponseWrapper$SaveContextServletOutputStream.flush(SaveContextOnUpdateOrErrorResponseWrapper.java:354) at com.sun.jersey.spi.container.servlet.WebComponent$Writer.flush(WebComponent.java:308) at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.flush(ContainerResponse.java:146) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) - locked <0x00075a610750> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at java.io.BufferedWriter.flush(BufferedWriter.java:254) - locked <0x00075a610750> (a java.io.OutputStreamWriter) at com.sun.jersey.core.util.ReaderWriter.writeToAsString(ReaderWriter.java:191) at com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:128) at com.sun.jersey.core.impl.provider.entity.StringProvider.writeTo(StringProvider.java:88) at com.sun.jersey.core.impl.provider.entity.StringProvider.writeTo(StringProvider.java:58) at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) This is during slow response times where many threads were opened (Tomcat configured with 500). Also, thread dump shows that 240 of tomcat threads are in following state which suggests that they are idle. "http-bio-28080-exec-1230" daemon prio=10 tid=0x7f812c361800 nid=0x29a0 waiting on condition [0x7f810a7e6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00071d193f50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Any clues would help. Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Idle Thread high CPU
On 3/10/16 5:23 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rallavagu, On 3/10/16 8:10 PM, Rallavagu wrote: On 3/10/16 2:33 PM, Christopher Schultz wrote: Rallavagu, On 3/10/16 5:16 PM, Rallavagu wrote: On 3/10/16 2:09 PM, Christopher Schultz wrote: Rallavagu, On 3/10/16 4:02 PM, Rallavagu wrote: On 3/10/16 11:54 AM, Christopher Schultz wrote: Are you sure you have matched-up the correct thread within the JVM that is using all that CPU? How are you measuring the CPU usage? It would the ID output from "top -H" mapping to "Native ID" in thread dump. My version of 'top' (Debian Linux) doesn't show thread ids. :( I seem to recall having to do some backflips to convert native thread id to Java thread id. Can you explain what you've done to do that? A typical top -H shows the following top - 11:40:11 up 190 days, 1:24, 1 user, load average: 5.74, 6.09, 5.78 Tasks: 759 total, 4 running, 755 sleeping, 0 stopped, 0 zombie Cpu(s): 18.4%us, 1.6%sy, 0.0%ni, 79.5%id, 0.1%wa, 0.0%hi, 0.5%si, 0.0%st Mem: 8057664k total, 7895252k used, 162412k free,63312k buffers Swap: 2064380k total, 199452k used, 1864928k free, 2125868k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 15648 tomcat20 0 9649m 4.8g 4520 R 87.3 62.6 7:24.24 java 21710 tomcat20 0 9649m 4.8g 4520 R 79.8 62.6 5:44.99 java 21694 tomcat20 0 9649m 4.8g 4520 S 74.3 62.6 5:39.40 java 7889 tomcat20 0 9649m 4.8g 4520 S 29.7 62.6 4:24.44 java 7878 tomcat20 0 9649m 4.8g 4520 S 27.8 62.6 4:36.82 java 21701 tomcat20 0 9649m 4.8g 4520 S 26.0 62.6 5:49.83 java After taking thread dump, I used Threadlogic which will show Native-ID as column which corresponds to PID shown above. https://java.net/projects/threadlogic This way it helps to determine the thread that might potentially causing high cpu. Okay. Are you serving a high rate of requests? It's possible that the thread is just doing a lot of (legitimate) work. The BIO connector is very basic: it uses blocking reads, and the thread dump you showed before showed it waiting on IO, so it should be completely idle, using no CPU time. It's *possible* that it's in a busy-wait state where it is performing a very short IO-wait in a loop that it never exits. But since you haven't specified any weird timeouts, etc. on your connector, I'm skeptical as to that being the cause. This thread stays at high CPU usage for quite a while? And every thread dump you do has the same (or very similar) stack trace? The symptom is that app is always high on CPU hovering between 75 - 85 and so looked at the thread dumps. Each thread dump shows few high cpu threads and some of those are supposedly idle. After looking at a particular thread by Id what it was doing in the previous thread some of these were reading on socket. So, that might be somewhat related to what you said about I/O wait. Also, if BIO is basic, what are other options? https://tomcat.apache.org/tomcat-8.0-doc/config/http.html#Standard_Imple mentation The NIO connector is more scalable, but the BIO should use the *least* resources when handling modest loads. I wasn't suggesting that BIO should be avoided due to its simplicity... quite the contrary, I was suggesting that the BIO connector *should* be well-behaved. Sounds good. So, the thread might be simply "winding down" from socket read? If many of those threads behaving the same way pushing CPU to high. Is there anything that I should be looking into or configure perhaps to address this or update to 7.0.68? - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlbiHf8ACgkQ9CaO5/Lv0PAHWQCbBHR4dIjpfv+K0qnJil50Buyk CIAAoKUByrZhri66vkU/OLdJ/01qMc2u =+9Fl -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Idle Thread high CPU
On 3/10/16 2:09 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rallavagu, On 3/10/16 4:02 PM, Rallavagu wrote: On 3/10/16 11:54 AM, Christopher Schultz wrote: Are you sure you have matched-up the correct thread within the JVM that is using all that CPU? How are you measuring the CPU usage? It would the ID output from "top -H" mapping to "Native ID" in thread dump. My version of 'top' (Debian Linux) doesn't show thread ids. :( I seem to recall having to do some backflips to convert native thread id to Java thread id. Can you explain what you've done to do that? A typical top -H shows the following top - 11:40:11 up 190 days, 1:24, 1 user, load average: 5.74, 6.09, 5.78 Tasks: 759 total, 4 running, 755 sleeping, 0 stopped, 0 zombie Cpu(s): 18.4%us, 1.6%sy, 0.0%ni, 79.5%id, 0.1%wa, 0.0%hi, 0.5%si, 0.0%st Mem: 8057664k total, 7895252k used, 162412k free,63312k buffers Swap: 2064380k total, 199452k used, 1864928k free, 2125868k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 15648 tomcat20 0 9649m 4.8g 4520 R 87.3 62.6 7:24.24 java 21710 tomcat20 0 9649m 4.8g 4520 R 79.8 62.6 5:44.99 java 21694 tomcat20 0 9649m 4.8g 4520 S 74.3 62.6 5:39.40 java 7889 tomcat20 0 9649m 4.8g 4520 S 29.7 62.6 4:24.44 java 7878 tomcat20 0 9649m 4.8g 4520 S 27.8 62.6 4:36.82 java 21701 tomcat20 0 9649m 4.8g 4520 S 26.0 62.6 5:49.83 java After taking thread dump, I used Threadlogic which will show Native-ID as column which corresponds to PID shown above. https://java.net/projects/threadlogic This way it helps to determine the thread that might potentially causing high cpu. Can you post your and/or configuration? Okay, so you are using the default (BIO) connector with no special configuration. I see you are running Tomcat 7.0.47. Could you please re-test with latest 7.0.68? Your version is 2.5 years old and there may be a performance bug fixed. Sure. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlbh8LQACgkQ9CaO5/Lv0PA89wCdHBDkMdKa/9JkNLGY80Ygrl0/ NpkAoL5kQ8cx/Qr9WcOpdHw3DxvurcrD =Ckbq -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Idle Thread high CPU
On 3/10/16 1:02 PM, Konstantin Kolinko wrote: 2016-03-10 22:54 GMT+03:00 Christopher Schultz <ch...@christopherschultz.net>: Rallavagu, On 3/10/16 2:11 PM, Rallavagu wrote: From a thread dump and corresponding "top" output it is reported that the following thread is consuming significant CPU (around 80%) "http-bio-28080-exec-437" daemon prio=10 tid=0x7f4acc0de000 nid=0x54ce waiting on condition [0x7f4b038f7000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00071846da50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject. awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java :467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.jav a:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java:615) at java.lang.Thread.run(Thread.java:744) Considering it is a thread from tomcat's thread pool and it suggests that it is "java.lang.Thread.State: TIMED_WAITING (parking)" wonder why this shows up as consuming high CPU. Any clues would be highly appreciated. Thanks in advance. Are you sure you have matched-up the correct thread within the JVM that is using all that CPU? How are you measuring the CPU usage? Can you post your and/or configuration? Also Sorry for not providing this information upfront. 1. Tomcat version =? 7.0.47 2. Java version =? Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) 3. You need to take several (e.g. 3) thread dumps to see whether there is some progress /change of state. There is some progress. But, similar but different thread(s) show high cpu in some of those. The last similar thread in search results for users@ list is in May 2015 "High cpu on Tomcat 8", a year ago. As there were no such threads recently, I think that bug has already been fixed. The only Tomcat code in that stacktrace is org.apache.tomcat.util.threads.TaskQueue. The connector type can be guessed as "bio" from the thread name, but as a thread name can be configured on an Executor, this identification is not reliable. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Idle Thread high CPU
On 3/10/16 11:54 AM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rallavagu, On 3/10/16 2:11 PM, Rallavagu wrote: From a thread dump and corresponding "top" output it is reported that the following thread is consuming significant CPU (around 80%) "http-bio-28080-exec-437" daemon prio=10 tid=0x7f4acc0de000 nid=0x54ce waiting on condition [0x7f4b038f7000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00071846da50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject. awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java :467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.jav a:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java:615) at java.lang.Thread.run(Thread.java:744) Considering it is a thread from tomcat's thread pool and it suggests that it is "java.lang.Thread.State: TIMED_WAITING (parking)" wonder why this shows up as consuming high CPU. Any clues would be highly appreciated. Thanks in advance. Are you sure you have matched-up the correct thread within the JVM that is using all that CPU? How are you measuring the CPU usage? It would the ID output from "top -H" mapping to "Native ID" in thread dump. Can you post your and/or configuration? Thanks - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlbh0Q0ACgkQ9CaO5/Lv0PDDpACfbmueOC3FIVAImoIY6P0GYgla iUUAnRimWHnwhvNR+q2KKyypxnFcoHBi =o7Ac -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Idle Thread high CPU
All, From a thread dump and corresponding "top" output it is reported that the following thread is consuming significant CPU (around 80%) "http-bio-28080-exec-437" daemon prio=10 tid=0x7f4acc0de000 nid=0x54ce waiting on condition [0x7f4b038f7000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00071846da50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Considering it is a thread from tomcat's thread pool and it suggests that it is "java.lang.Thread.State: TIMED_WAITING (parking)" wonder why this shows up as consuming high CPU. Any clues would be highly appreciated. Thanks in advance. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Socket Read long running
On 1/21/16 4:13 AM, Christopher Schultz wrote: Rallavagu, On 1/19/16 6:14 PM, Rallavagu wrote: On 1/19/16 2:43 PM, Mark Thomas wrote: On 19/01/2016 22:36, Rallavagu wrote: Also, it could be keep-alive for client connection as well. In any case, how long a keep-alive connection will be in this state by default? Thanks. This behaviour is entirely normal. Why are you concerned about it? I was analyzing thread dump as the application experiences sudden high response times and eventually becomes normal. Regarding how long the thread will be in this state, the default keep-alive timeout for the HTTP BIO connector can be found in the documentation. (Yes, I do happen to know what it is but think of this as an exercise for the reader.) From the documentation keepAliveTimeout defaults to connectionTimeout. "The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite) timeout. The default value is 6 (i.e. 60 seconds) but note that the standard server.xml that ships with Tomcat sets this to 2 (i.e. 20 seconds). Unless disableUploadTimeout is set to false, this timeout will also be used when reading the request body (if any)." You might want to consider switching to an NIO-based connector. The NIO-based connectors do not block a request-processing thread during the keep-alive wait time, so fewer threads can handle the same number of actual incoming requests (instead of waiting-around potentially doing no additional work). You may still have a response-time problem after that, but it won't be due to the threading model. If your load-balancer configured to maintain keep-alive connections to your Tomcat instance(s)? If so, what are the details of that configuration? They have configured for a pool of 50 keep-alive connections with 10 seconds timeout. Thanks -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Socket Read long running
I have this long running thread. It appears to be reading but the stack trace does not give much of a clue. Could anyone help with where to start? Thanks. "tomcat-exec-2655" daemon prio=10 tid=0x7fc459061000 nid=0x6a58 runnable [0x7fc4a67e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:991) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked <0x000799d09cb8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00072eebc6c0> (a java.util.concurrent.ThreadPoolExecutor$Worker) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Socket Read long running
Also, it could be keep-alive for client connection as well. In any case, how long a keep-alive connection will be in this state by default? Thanks. On 1/19/16 2:24 PM, Rallavagu wrote: Thanks Mark. It seems to be running for almost 10 seconds and there is a Load Balancer between. Is it a suspect? On 1/19/16 2:09 PM, Mark Thomas wrote: On 19/01/2016 21:42, Rallavagu wrote: I have this long running thread. It appears to be reading but the stack trace does not give much of a clue. Could anyone help with where to start? Thanks. Looks like an HTTP keep-alive connection waiting for the next request. Mark Tomcat 7.0.42 with JDK 7 "tomcat-exec-2655" daemon prio=10 tid=0x7fc459061000 nid=0x6a58 runnable [0x7fc4a67e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:991) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked <0x000799d09cb8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00072eebc6c0> (a java.util.concurrent.ThreadPoolExecutor$Worker) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Socket Read long running
Thanks Mark. It seems to be running for almost 10 seconds and there is a Load Balancer between. Is it a suspect? On 1/19/16 2:09 PM, Mark Thomas wrote: On 19/01/2016 21:42, Rallavagu wrote: I have this long running thread. It appears to be reading but the stack trace does not give much of a clue. Could anyone help with where to start? Thanks. Looks like an HTTP keep-alive connection waiting for the next request. Mark Tomcat 7.0.42 with JDK 7 "tomcat-exec-2655" daemon prio=10 tid=0x7fc459061000 nid=0x6a58 runnable [0x7fc4a67e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:991) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked <0x000799d09cb8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00072eebc6c0> (a java.util.concurrent.ThreadPoolExecutor$Worker) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Socket Read long running
I have this long running thread. It appears to be reading but the stack trace does not give much of a clue. Could anyone help with where to start? Thanks. Tomcat 7.0.42 with JDK 7 "tomcat-exec-2655" daemon prio=10 tid=0x7fc459061000 nid=0x6a58 runnable [0x7fc4a67e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:991) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked <0x000799d09cb8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00072eebc6c0> (a java.util.concurrent.ThreadPoolExecutor$Worker) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Socket Read long running
On 1/19/16 2:43 PM, Mark Thomas wrote: On 19/01/2016 22:36, Rallavagu wrote: Also, it could be keep-alive for client connection as well. In any case, how long a keep-alive connection will be in this state by default? Thanks. This behaviour is entirely normal. Why are you concerned about it? I was analyzing thread dump as the application experiences sudden high response times and eventually becomes normal. Regarding how long the thread will be in this state, the default keep-alive timeout for the HTTP BIO connector can be found in the documentation. (Yes, I do happen to know what it is but think of this as an exercise for the reader.) From the documentation keepAliveTimeout defaults to connectionTimeout. "The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite) timeout. The default value is 6 (i.e. 60 seconds) but note that the standard server.xml that ships with Tomcat sets this to 2 (i.e. 20 seconds). Unless disableUploadTimeout is set to false, this timeout will also be used when reading the request body (if any)." Thanks Mark On 1/19/16 2:24 PM, Rallavagu wrote: Thanks Mark. It seems to be running for almost 10 seconds and there is a Load Balancer between. Is it a suspect? On 1/19/16 2:09 PM, Mark Thomas wrote: On 19/01/2016 21:42, Rallavagu wrote: I have this long running thread. It appears to be reading but the stack trace does not give much of a clue. Could anyone help with where to start? Thanks. Looks like an HTTP keep-alive connection waiting for the next request. Mark Tomcat 7.0.42 with JDK 7 "tomcat-exec-2655" daemon prio=10 tid=0x7fc459061000 nid=0x6a58 runnable [0x7fc4a67e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:168) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:991) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked <0x000799d09cb8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00072eebc6c0> (a java.util.concurrent.ThreadPoolExecutor$Worker) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ClientAbortException: java.io.IOException: Failed to send AJP message
This usually means that "client" has disconnected before the request could be completed. Generally, this might happen when a user navigates away from a web page before it is completely rendered. You might want to gather more information to understand this better. On 10/26/15 7:15 AM, Yogesh Patel wrote: In out application we are getting following error: org.apache.catalina.core.StandardWrapperValve.invoke:Line 211 - ClientAbortException: java.io.IOException: Failed to send AJP message - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tracking down memory leak
Please take a look at Memory Analyzer tool (http://www.eclipse.org/mat/). Run the app and take the heap dump while app is running and use the tool to analyze it. You could use VisualVM with plugins to get instrumentation or you could use hprof (http://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html) HTH On 10/20/15 6:20 AM, David kerber wrote: I'm trying to track down the source of a memory leak in one of my applications. I have examined the code but have been unable to fix it, so am looking for some way of instrumenting my app while running on the server. What is the easiest/best (I realize those two criteria may not give the same answer!) way? Running TC 8.0.20-something, JRE 8.0.something recent, on windows Server 2012 R2. Thanks for any suggestions! Dave - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 7 http-bio-8080-acceptor
Tomcat 7.0.14 Looking into a thread dump (after tomcat stopped accepting connections) http-bio-8080-acceptor is missing while http-bio-8080-AsyncTimeout thread is present. The application is deployed to a non root context while ROOT is served by tomcat's default ROOT app. What could have caused this behavior? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: BLOCKED threads
Logs are written to local disk. Don't have data on disk stats. Thanks. On May 5, 2014, at 7:19 AM, Daniel Mikusa dmik...@gopivotal.com wrote: On May 3, 2014, at 9:08 PM, Rallavagu rallav...@gmail.com wrote: Here is the thread BLOCKED waiting on another lock. http-bio-28080-exec-613 daemon prio=10 tid=0x7fcbac0e0800 nid=0x7897 waiting for monitor entry [0x7fcb915d3000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.logging.StreamHandler.publish(StreamHandler.java:191) - waiting to lock 0x0007008187b0 (a java.util.logging.ConsoleHandler) at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105) at java.util.logging.Logger.log(Logger.java:610) at java.util.logging.Logger.doLog(Logger.java:631) at java.util.logging.Logger.logp(Logger.java:831) at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185) at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) - locked 0x0007e1b0f808 (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Looks like this is waiting to log something. Where are you writing your logs? Is it a local disk or network disk? Is the disk busy or otherwise slow? Dan Locked ownable synchronizers: - 0x0007e1b0f8d0 (a java.util.concurrent.ThreadPoolExecutor$Worker) On 5/2/14, 9:19 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Rallavagu, On 5/2/14, 6:22 PM, Rallavagu wrote: Tomcat Version: 7.0.47 JVM Version: 1.7.0_51-b13 I see many blocked threads (90) in the thread dump. There are mainly two monitors that block 69 threads. One of them is below. It appears that it is simply trying to log. -- http-bio-28080-exec-396 daemon prio=10 tid=0x7fcbc814f000 nid=0x5804 runnable [0x7fcc2144d000] java.lang.Thread.State: RUNNABLE at java.lang.Throwable.getStackTraceElement(Native Method) This thread is not blocked. What makes you think it is? at java.lang.Throwable.getOurStackTrace(Throwable.java:827) - locked 0x0007e1886340 (a java.util.NoSuchElementException) at java.lang.Throwable.printStackTrace(Throwable.java:656) - locked 0x0007e207a5a8 (a java.io.PrintWriter) at java.lang.Throwable.printStackTrace(Throwable.java:721) at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157) - locked 0x0007008187e8 (a java.util.logging.SimpleFormatter) at java.util.logging.StreamHandler.publish(StreamHandler.java:196) - locked 0x0007008187b0 (a java.util.logging.ConsoleHandler) at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105) at java.util.logging.Logger.log(Logger.java:610) at java.util.logging.Logger.doLog(Logger.java:631) at java.util.logging.Logger.logp(Logger.java:831) at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185) at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408
Re: BLOCKED threads
Chris, Thanks for the reply. Sorry for not being very clear. Actually, those two threads holding the locks that caused other threads to be blocked. Here is one of the threads that was blocked because of lock on StandarClassLoader. There are around 47 threads in BLOCKED state waiting on the same lock. http-bio-28080-exec-548 daemon prio=10 tid=0x7fcbac06 nid=0x76e0 waiting for monitor entry [0x7fcc012ef000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.ClassLoader.loadClass(ClassLoader.java:405) - waiting to lock 0x000700810fc8 (a org.apache.catalina.loader.StandardClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at com.singularity.ee.agent.util.be.a(be.java:19) at com.singularity.ee.agent.appagent.services.transactionmonitor.jdbc.hb.f(hb.java:311) at com.singularity.ee.agent.appagent.services.transactionmonitor.jdbc.hb.c(hb.java:260) at com.singularity.ee.agent.appagent.services.transactionmonitor.jdbc.hb.b(hb.java:144) at com.singularity.ee.agent.appagent.services.transactionmonitor.jdbc.c.a(c.java:76) at com.singularity.ee.agent.appagent.services.transactionmonitor.common.e.a(e.java:488) at com.singularity.ee.agent.appagent.services.transactionmonitor.common.e.a(e.java:441) at com.singularity.ee.agent.appagent.services.transactionmonitor.common.e.a(e.java:375) at com.singularity.ee.agent.appagent.services.transactionmonitor.jdbc.c.a(c.java:32) at com.singularity.ee.agent.appagent.services.bciengine.a.onMethodEnd(a.java:62) at com.singularity.ee.agent.appagent.entrypoint.bciengine.FastMethodInterceptorDelegator.safeOnMethodEndNoReentrantCheck(FastMethodInterceptorDelegator.java:408) at com.singularity.ee.agent.appagent.entrypoint.bciengine.FastMethodInterceptorDelegator.safeOnMethodEnd(FastMethodInterceptorDelegator.java:350) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491) at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:122) at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81) at com.sun.proxy.$Proxy397.executeQuery(Unknown Source) at org.hibernate.loader.Loader.getResultSet(Loader.java:2031) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1832) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1811) at org.hibernate.loader.Loader.doQuery(Loader.java:899) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:341) at org.hibernate.loader.Loader.doList(Loader.java:2516) at org.hibernate.loader.Loader.doList(Loader.java:2502) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2332) at org.hibernate.loader.Loader.list(Loader.java:2327) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:490) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:355) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1247) at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101) On 5/2/14, 9:19 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Rallavagu, On 5/2/14, 6:22 PM, Rallavagu wrote: Tomcat Version: 7.0.47 JVM Version: 1.7.0_51-b13 I see many blocked threads (90) in the thread dump. There are mainly two monitors that block 69 threads. One of them is below. It appears that it is simply trying to log. -- http-bio-28080-exec-396 daemon prio=10 tid=0x7fcbc814f000 nid=0x5804 runnable [0x7fcc2144d000] java.lang.Thread.State: RUNNABLE at java.lang.Throwable.getStackTraceElement(Native Method) This thread is not blocked. What makes you think it is? at java.lang.Throwable.getOurStackTrace(Throwable.java:827) - locked 0x0007e1886340 (a java.util.NoSuchElementException) at java.lang.Throwable.printStackTrace(Throwable.java:656) - locked 0x0007e207a5a8 (a java.io.PrintWriter) at java.lang.Throwable.printStackTrace(Throwable.java:721) at java.util.logging.SimpleFormatter.format
Re: BLOCKED threads
Here is the thread BLOCKED waiting on another lock. http-bio-28080-exec-613 daemon prio=10 tid=0x7fcbac0e0800 nid=0x7897 waiting for monitor entry [0x7fcb915d3000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.logging.StreamHandler.publish(StreamHandler.java:191) - waiting to lock 0x0007008187b0 (a java.util.logging.ConsoleHandler) at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105) at java.util.logging.Logger.log(Logger.java:610) at java.util.logging.Logger.doLog(Logger.java:631) at java.util.logging.Logger.logp(Logger.java:831) at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185) at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) - locked 0x0007e1b0f808 (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Locked ownable synchronizers: - 0x0007e1b0f8d0 (a java.util.concurrent.ThreadPoolExecutor$Worker) On 5/2/14, 9:19 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Rallavagu, On 5/2/14, 6:22 PM, Rallavagu wrote: Tomcat Version: 7.0.47 JVM Version: 1.7.0_51-b13 I see many blocked threads (90) in the thread dump. There are mainly two monitors that block 69 threads. One of them is below. It appears that it is simply trying to log. -- http-bio-28080-exec-396 daemon prio=10 tid=0x7fcbc814f000 nid=0x5804 runnable [0x7fcc2144d000] java.lang.Thread.State: RUNNABLE at java.lang.Throwable.getStackTraceElement(Native Method) This thread is not blocked. What makes you think it is? at java.lang.Throwable.getOurStackTrace(Throwable.java:827) - locked 0x0007e1886340 (a java.util.NoSuchElementException) at java.lang.Throwable.printStackTrace(Throwable.java:656) - locked 0x0007e207a5a8 (a java.io.PrintWriter) at java.lang.Throwable.printStackTrace(Throwable.java:721) at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157) - locked 0x0007008187e8 (a java.util.logging.SimpleFormatter) at java.util.logging.StreamHandler.publish(StreamHandler.java:196) - locked 0x0007008187b0 (a java.util.logging.ConsoleHandler) at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105) at java.util.logging.Logger.log(Logger.java:610) at java.util.logging.Logger.doLog(Logger.java:631) at java.util.logging.Logger.logp(Logger.java:831) at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185) at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) - locked 0x0007e0ba5dd8 (a org.apache.tomcat.util.net.SocketWrapper
BLOCKED threads
All, Tomcat Version: 7.0.47 JVM Version: 1.7.0_51-b13 I see many blocked threads (90) in the thread dump. There are mainly two monitors that block 69 threads. One of them is below. It appears that it is simply trying to log. -- http-bio-28080-exec-396 daemon prio=10 tid=0x7fcbc814f000 nid=0x5804 runnable [0x7fcc2144d000] java.lang.Thread.State: RUNNABLE at java.lang.Throwable.getStackTraceElement(Native Method) at java.lang.Throwable.getOurStackTrace(Throwable.java:827) - locked 0x0007e1886340 (a java.util.NoSuchElementException) at java.lang.Throwable.printStackTrace(Throwable.java:656) - locked 0x0007e207a5a8 (a java.io.PrintWriter) at java.lang.Throwable.printStackTrace(Throwable.java:721) at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157) - locked 0x0007008187e8 (a java.util.logging.SimpleFormatter) at java.util.logging.StreamHandler.publish(StreamHandler.java:196) - locked 0x0007008187b0 (a java.util.logging.ConsoleHandler) at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105) at java.util.logging.Logger.log(Logger.java:610) at java.util.logging.Logger.doLog(Logger.java:631) at java.util.logging.Logger.logp(Logger.java:831) at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185) at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) - locked 0x0007e0ba5dd8 (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -- The second one has the lock on StandardClassLoader. -- http-bio-28080-exec-605 daemon prio=10 tid=0x7fcbc82b8800 nid=0x77e6 runnable [0x7fcb919d6000] java.lang.Thread.State: RUNNABLE at java.lang.ClassLoader.findLoadedClass0(Native Method) at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1093) at java.lang.ClassLoader.loadClass(ClassLoader.java:407) - locked 0x000700810fc8 (a org.apache.catalina.loader.StandardClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2566) at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296) at java.util.ResourceBundle.getBundle(ResourceBundle.java:721) at oracle.net.ns.Message11.getMessage(Message11.java:62) at oracle.net.ns.NetException.getMessage(NetException.java:222) at oracle.net.ano.AnoComm.b(Unknown Source) at oracle.net.ano.AnoComm.o(Unknown Source) at oracle.net.ano.Ano.a(Unknown Source) at oracle.net.ano.Ano.negotiation(Unknown Source) at oracle.net.ns.NSProtocol.connect(NSProtocol.java:407) at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292) at oracle.jdbc.driver.PhysicalConnection.init(PhysicalConnection.java:508) at oracle.jdbc.driver.T4CConnection.init(T4CConnection.java:203) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33) at
