Hello Everyone, During database failover resiliency testing we noticed something unusual which we still cannot explain. So, reaching out to the community for help.
Application Setup: AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016 Observations: After database failover, we noticed that HTTP thread pool size jumped from 10 to 50. No. of busy threads jumped to 40 for a minute and then came down to normal range(1-5). But thread pool size remained 50 for the next one hour. When we collected the heap dump after one hour, thread pool size jumped again to ~90. Thread dump analysis is showing most of the HTTP threads were just waiting to get some work[1]. That looked surprising to me. Few of them are runnable and doing some work - which matches "busy thread" count. During all this timeframe load on the system remained the same. We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We do not have any keepAliveTimeout set in Tomcat. App uses session affinity. We were looking for some connector properties which deal with idle threads but found none[3]. We are not seeing any "problem" in our app after DB failover but this is something also newer to us and we are trying to explain it. Tomcat's memory usage increased by a few MB as new threads were spawned. My expectation was that if threads in the thread pool are not getting used and idle, after sometime thread pool size will reduce to "minSpareThreads" size which is 10 in our case. Is that expectation wrong? Can you please share your thoughts? [1] "http-apr-18110-exec-63" #2170 daemon prio=5 os_prio=0 tid=0x0000000045f03000 nid=0x30d4 waiting on condition [0x000000005a4cf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006a5400178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - None [2] <Connector port="18110" protocol="HTTP/1.1" maxThreads="500" acceptCount="300" URIEncoding="UTF-8" redirectPort="18113" connectionTimeout="30000"/> <Connector port="18113" threadPriority="6" compression="force" compressionMinSize="2048" compressableMimeTypes="text/html,text/xml,text/plain,application/json" sslProtocol="TLS" secure="true" scheme="https" protocol="HTTP/1.1" maxThreads="150" maxSavePostSize="-1" maxHttpHeaderSize="8192" keystoreFile="conf/.keystore" enableLookups="false" disableUploadTimeout="true" clientAuth="false" acceptCount="100" SSLPassword="" SSLEnabled="true" SSLCertificateKeyFile="\app\my.key" SSLCertificateFile="\app\mycert.cer"/> [3] https://tomcat.apache.org/tomcat-7.0-doc/config/http.html Thanks & Regards, Suvendu --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
