Re: Tomcat - threads / throughput limits?
On 08/11/2007, at 4:51 PM, Jim Cox wrote: On Nov 8, 2007 10:41 AM, [EMAIL PROTECTED] wrote: In resolving our current bottleneck i used JProfiler to see what the tomcat applications were doing and when under high load there are a lot of threads which are blocked on this: org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() This suggests that they are spending all their time waiting for tomcat to run them, either way they're not runnable anyway. http80-Processor131 daemon prio=5 tid=0x00baa7c0 nid=0x10f9 in Object.wait() [0xa5e81000..0xa5e819c8] at java.lang.Object.wait(Native Method) - waiting on 0xc9d99d20 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) FWIW, I always interpreted those as HTTP servicing threads waiting for something to do (i.e. not a bad thing at all). Have you tried running your profiler against a tomcat not doing ANY requests? I agree with Jim on this one - These are threads WAITING on a connection... and not doing anything. Have a look in tomcat manager, and then you can see how many 'free' connections you have. Cheers Andrew smime.p7s Description: S/MIME cryptographic signature
Re: Tomcat - threads / throughput limits?
It doesn't answer your question in general, but I increased mine from 300 to 900 threads on a dual-processor, dual-core machine, and saw significantly increased throughput with short, simple requests. I didn't try any in-between settings, though, so I don't know if there might be a sweet spot somewhere between those settings, and I don't know if going higher would help or not, because the setting it's at right now handles the full load with no waiting. D [EMAIL PROTECTED] wrote: Hi, We have an architecture where we have many tomcat app servers load balanced by apache at the front. In resolving our current bottleneck i used JProfiler to see what the tomcat applications were doing and when under high load there are a lot of threads which are blocked on this: org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() This suggests that they are spending all their time waiting for tomcat to run them, either way they're not runnable anyway. Our app pushes through huge quantites of very quick requests - typically each request takes a total of 50ms. We have a limit of 300 connections on our AJP connector. The point of blocked threads occurs way before we hit that limit. At the same time neither the machine is busy, nor are there any obvious problems with memory usage for the instance. And indeed; Adding more app servers keeps the load to each one down and does resolve the problem - i.e. we can increase throughput. So; My question is, is there a reccomended limit to how many threads tomcat can handle before it spends more time managing threads than it does doing real work? Thanks! Dan - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat - threads / throughput limits?
On Nov 8, 2007 10:41 AM, [EMAIL PROTECTED] wrote: Hi, We have an architecture where we have many tomcat app servers load balanced by apache at the front. In resolving our current bottleneck i used JProfiler to see what the tomcat applications were doing and when under high load there are a lot of threads which are blocked on this: org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() This suggests that they are spending all their time waiting for tomcat to run them, either way they're not runnable anyway. Our app pushes through huge quantites of very quick requests - typically each request takes a total of 50ms. We have a limit of 300 connections on our AJP connector. The point of blocked threads occurs way before we hit that limit. At the same time neither the machine is busy, nor are there any obvious problems with memory usage for the instance. And indeed; Adding more app servers keeps the load to each one down and does resolve the problem - i.e. we can increase throughput. So; My question is, is there a reccomended limit to how many threads tomcat can handle before it spends more time managing threads than it does doing real work? Thanks! Dan Not quite answering your question, but re your blocked threads, do you mean something like the following in a stack dump? http80-Processor131 daemon prio=5 tid=0x00baa7c0 nid=0x10f9 in Object.wait() [0xa5e81000..0xa5e819c8] at java.lang.Object.wait(Native Method) - waiting on 0xc9d99d20 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:429) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:660) - locked 0xc9d99d20 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:534) FWIW, I always interpreted those as HTTP servicing threads waiting for something to do (i.e. not a bad thing at all). - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Tomcat - threads / throughput limits?
Hi, We have an architecture where we have many tomcat app servers load balanced by apache at the front. In resolving our current bottleneck i used JProfiler to see what the tomcat applications were doing and when under high load there are a lot of threads which are blocked on this: org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() This suggests that they are spending all their time waiting for tomcat to run them, either way they're not runnable anyway. Our app pushes through huge quantites of very quick requests - typically each request takes a total of 50ms. We have a limit of 300 connections on our AJP connector. The point of blocked threads occurs way before we hit that limit. At the same time neither the machine is busy, nor are there any obvious problems with memory usage for the instance. And indeed; Adding more app servers keeps the load to each one down and does resolve the problem - i.e. we can increase throughput. So; My question is, is there a reccomended limit to how many threads tomcat can handle before it spends more time managing threads than it does doing real work? Thanks! Dan - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat - threads / throughput limits?
Have you tried running your profiler against a tomcat not doing ANY requests? I agree with Jim on this one - These are threads WAITING on a connection... and not doing anything. Have a look in tomcat manager, and then you can see how many 'free' connections you have. We only see the blocked threads when the instance is under heavy load. When the instances is clean and newly started most threads are idel. when it's under medium load you see a few are processing and the vast majority of the rest are all in net i/o Then under very heavy load you start to see the blocking... Well thats what we see anyway! - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
