Re: HTTP thread pool size remained high after database failover
Thank you Chris for your response. On Wed, Nov 11, 2020 at 10:02 PM Christopher Schultz wrote: > > Suvendu, > > On 11/11/20 06:40, Suvendu Sekhar Mondal wrote: > > Application Setup: > > AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server > > 2016 > > As Michael pointed-out, these all need to be upgraded, but they are not > affecting your experience, here. > > > Observations: > > After database failover, we noticed that HTTP thread pool size jumped > > from 10 to 50. No. of busy threads jumped to 40 for a minute and then > > came down to normal range(1-5). But thread pool size remained 50 for > > the next one hour. When we collected the heap dump after one hour, > > thread pool size jumped again to ~90. Thread dump analysis is showing > > most of the HTTP threads were just waiting to get some work[1]. That > > looked surprising to me. Few of them are runnable and doing some work > > - which matches "busy thread" count. During all this timeframe load on > > the system remained the same. > > > > We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We > > do not have any keepAliveTimeout set in Tomcat. App uses session > > affinity. We were looking for some connector properties which deal > > with idle threads but found none[3]. > > You won't find them. > Okay. > > We are not seeing any "problem" in our app after DB failover but this > > is something also newer to us and we are trying to explain it. > > What are you trying to explain? > We are simply trying to understand this behavior as this is something new we noticed. > > Tomcat's memory usage increased by a few MB as new threads were > > spawned. My expectation was that if threads in the thread pool are not > > getting used and idle, after sometime thread pool size will reduce to > > "minSpareThreads" size which is 10 in our case. Is that expectation > > wrong? Can you please share your thoughts? > > This is a common misconception. If you do not configure an "executor" > then Tomcat will create one for you and it will NOT remove idle threads > from the thread pool. > > The solution is to manually-specify an Executor: > > minSpareThreads="Y" /> > > > > With this configuration, you should see your thread count stay between Y > and X, but when X are not necessary, your thread count should drop > towards Y as appropriate. > Thanks for the explanation. This is the kind of information I was looking for in Tomcat docs but was not able to locate it. We will try this approach and let you know the outcome. > Hope that helps, > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP thread pool size remained high after database failover
Suvendu, On 11/11/20 06:40, Suvendu Sekhar Mondal wrote: Application Setup: AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016 As Michael pointed-out, these all need to be upgraded, but they are not affecting your experience, here. Observations: After database failover, we noticed that HTTP thread pool size jumped from 10 to 50. No. of busy threads jumped to 40 for a minute and then came down to normal range(1-5). But thread pool size remained 50 for the next one hour. When we collected the heap dump after one hour, thread pool size jumped again to ~90. Thread dump analysis is showing most of the HTTP threads were just waiting to get some work[1]. That looked surprising to me. Few of them are runnable and doing some work - which matches "busy thread" count. During all this timeframe load on the system remained the same. We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We do not have any keepAliveTimeout set in Tomcat. App uses session affinity. We were looking for some connector properties which deal with idle threads but found none[3]. You won't find them. We are not seeing any "problem" in our app after DB failover but this is something also newer to us and we are trying to explain it. What are you trying to explain? Tomcat's memory usage increased by a few MB as new threads were spawned. My expectation was that if threads in the thread pool are not getting used and idle, after sometime thread pool size will reduce to "minSpareThreads" size which is 10 in our case. Is that expectation wrong? Can you please share your thoughts? This is a common misconception. If you do not configure an "executor" then Tomcat will create one for you and it will NOT remove idle threads from the thread pool. The solution is to manually-specify an Executor: minSpareThreads="Y" /> With this configuration, you should see your thread count stay between Y and X, but when X are not necessary, your thread count should drop towards Y as appropriate. Hope that helps, -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP thread pool size remained high after database failover
Hi Michael, On Wed, Nov 11, 2020 at 6:03 PM Michael Osipov wrote: > > Am 2020-11-11 um 12:40 schrieb Suvendu Sekhar Mondal: > > Hello Everyone, > > > > During database failover resiliency testing we noticed something > > unusual which we still cannot explain. So, reaching out to the > > community for help. > > > > Application Setup: > > AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server > > 2016 > > Almost of them them are outdated, you should upgrade first. I know. We will be upgrading them in the next release. For this release we have to use aforementioned softwares. For the last three releases we used the same softwares but had not noticed this issue - maybe we were not looking that closely. Now, we are seeing this and can't let it go without explaining it. :) So, any thoughts on this? > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP thread pool size remained high after database failover
Am 2020-11-11 um 12:40 schrieb Suvendu Sekhar Mondal: Hello Everyone, During database failover resiliency testing we noticed something unusual which we still cannot explain. So, reaching out to the community for help. Application Setup: AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016 Almost of them them are outdated, you should upgrade first. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
HTTP thread pool size remained high after database failover
Hello Everyone, During database failover resiliency testing we noticed something unusual which we still cannot explain. So, reaching out to the community for help. Application Setup: AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016 Observations: After database failover, we noticed that HTTP thread pool size jumped from 10 to 50. No. of busy threads jumped to 40 for a minute and then came down to normal range(1-5). But thread pool size remained 50 for the next one hour. When we collected the heap dump after one hour, thread pool size jumped again to ~90. Thread dump analysis is showing most of the HTTP threads were just waiting to get some work[1]. That looked surprising to me. Few of them are runnable and doing some work - which matches "busy thread" count. During all this timeframe load on the system remained the same. We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We do not have any keepAliveTimeout set in Tomcat. App uses session affinity. We were looking for some connector properties which deal with idle threads but found none[3]. We are not seeing any "problem" in our app after DB failover but this is something also newer to us and we are trying to explain it. Tomcat's memory usage increased by a few MB as new threads were spawned. My expectation was that if threads in the thread pool are not getting used and idle, after sometime thread pool size will reduce to "minSpareThreads" size which is 10 in our case. Is that expectation wrong? Can you please share your thoughts? [1] "http-apr-18110-exec-63" #2170 daemon prio=5 os_prio=0 tid=0x45f03000 nid=0x30d4 waiting on condition [0x5a4cf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0006a5400178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - None [2] [3] https://tomcat.apache.org/tomcat-7.0-doc/config/http.html Thanks & Regards, Suvendu - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org