Re: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status
Christopher Schultz-2 wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kapilok, On 4/6/2010 11:19 AM, kapilok wrote: Does Tomcat guarantee recovery in case of this Error? The JVM/Tomcat should recover in the sense that service threads that complete their work should go back into the thread pool and accept new requests. Unfortunately, your webapp can significantly impact the ability for those threads to complete their request handling. Can you point me to the documentation? There is no documentation that covers this recovery: it's just a feature of using thread pools. Please suggest a way to test / validate the recovery. First, figure out what the real problem is. The symptom is obvious (see your subject line). To see what those threads are doing, allow this situation to occur, then do the following: 1. Perform whatever magic you need to solve the database overload issue 2. Wait a minute or two to give your webapp a chance to breathe 3. Perform a thread dump on your webapp (http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F) 4. Wait 30 seconds 5. Perform another thread dump on your webapp 6. Inspect the thread dump(s) to see what your threads are stuck doing. If they're stuck in your webapp's code, go fix that and try again. If they're stuck in Tomcat's code, post back and we'll take a look. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku7WkgACgkQ9CaO5/Lv0PA/rgCffQZFnykd/0eXKdbOxlY/6XA9 TskAoLNKqjjcJbrAyoKeqT5J6TRnsVKX =KvA6 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Chris, I think the thread dump does suggest that it is the webapp code (c3p0 connection pool) and not tomcat. But Please take a look anyways: (pasted partial dump only) Here's my test scenario: 1. Start Tomcat (with maxThreads=40 ) 2. Run JMeter Load (40 concurrent with some ramp up) - All requests succeed 3. Now load the database with some heavy process, so CPU consumption is high 4. Run same jMeter load; now response times are slow 5. Get ThreadPool Full Error; Take Thread Dump 6. Kill all JMeter requests; so Tomcat can breathe 7. Wait 10 minutes. Try login to webapp - Cant login. Browser does not display Login page or any other page. 8. Take another Thread Dump. Its the same. NOTE: Main Thread and TP-Processor4 are still running (observed with VisualVM1.1 on JDK 6) 9. Try running jMeter load. No reponse - No requests going through. 10. Waited 30 minutes and had no option but to bounce Tomcat. Bottom Line: What can I do so I don't have to bounce Tomcat when I run into this situation? Partial Thread Dump *** http-9080-Processor40 daemon prio=6 tid=0x4d1ba000 nid=0x1e0 in Object.wait() [0x5162e000..0x5162fd94] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x07b12648 (a com.mchange.v2.resourcepool.BasicResourcePool) at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315) at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557) - locked 0x07b12648 (a com.mchange.v2.resourcepool.BasicResourcePool) at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525) at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113) at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:523) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:587) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:612) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:644) ... com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2 daemon prio=6 tid=0x4dcda400 nid=0x6ac in Object.wait() [0x5068f000..0x5068fb94] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0a779ea8 (a com.mchange.v2.async.ThreadPoolAsynchronousRunner) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534) - locked 0x0a779ea8 (a com.mchange.v2.async.ThreadPoolAsynchronousRunner) Locked ownable synchronizers: -
Re: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kapilok, On 4/9/2010 9:46 AM, kapilok wrote: 1. Start Tomcat (with maxThreads=40 ) 2. Run JMeter Load (40 concurrent with some ramp up) - All requests succeed Good. 3. Now load the database with some heavy process, so CPU consumption is high 4. Run same jMeter load; now response times are slow This should be no surprise, as CPU time be scarce. 5. Get ThreadPool Full Error; Take Thread Dump This also shouldn't be a surprise: with maxThreads=40, you can only handle 40 simultaneous connections. If you have 1+ heavy process taking connections plus your 40 incoming ones, some will be denied. Or, are you saying that you are running a CPU-intensive process outside of Tomcat on the db server? 6. Kill all JMeter requests; so Tomcat can breathe 7. Wait 10 minutes. Try login to webapp - Cant login. Browser does not display Login page or any other page. Sounds like deadlock. 8. Take another Thread Dump. Its the same. NOTE: Main Thread and TP-Processor4 are still running (observed with VisualVM1.1 on JDK 6) 9. Try running jMeter load. No reponse - No requests going through. 10. Waited 30 minutes and had no option but to bounce Tomcat. Yep, sounds like deadlock to me. Bottom Line: What can I do so I don't have to bounce Tomcat when I run into this situation? If you really have deadlocked your system, you have no choice but to restart Tomcat. Are you using a JDBC connection pool? If so, what kind, and under what configuration? Partial Thread Dump *** http-9080-Processor40 daemon prio=6 tid=0x4d1ba000 nid=0x1e0 in Object.wait() [0x5162e000..0x5162fd94] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x07b12648 (a com.mchange.v2.resourcepool.BasicResourcePool) [...] org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:523) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:587) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:612) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:644) ... If the above thread never makes any progress, then it's likely that you have exhausted your connection pool. Usually this situation would correct itself, because one (or more) connections would eventually be returned to the pool, and become available for other clients. If this is a deadlock scenario centering around the JDBC connection pool, then you likely have a situation in your code where you do this: Connection c1 = pool.getConnection(); ... Connection c2 = pool.getConnection(); The above code can cause deadlock because you can end up obtaining one connection and then waiting forever for the second one (meanwhile other threads are also waiting forever for their second connections). com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2 daemon prio=6 tid=0x4dcda400 nid=0x6ac in Object.wait() [0x5068f000..0x5068fb94] java.lang.Thread.State: TIMED_WAITING (on object monitor) All of these threads look like they are waiting to do work that hasn't yet been assigned. These threads look good to me. I'm not sure how you're using the Spring framework to do your SQL stuff for you, but you should check to see that you aren't obtaining a connection and then firing-off another query /without/ using that connection: that will put you into the situation above. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku/RuYACgkQ9CaO5/Lv0PB50ACgtyqlmAckBoQXd9Jhzu0d1osA rxEAoJM1JEuL/HHvaFOl0KF/OAwVnDUK =36uP -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status
I need to restart tomcat after the error: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status This happens when the Database gets overloaded and responds slowly. I would like tomcat to recover (without having to restart), once I have solved the database overload issue (which takes a few minutes). I am using Tomcat 5.5.20 on Windows Server 2003. Please advise. Thanks, Kapil -- View this message in context: http://old.nabble.com/Recover-from-SEVERE%3A-All-threads-%28400%29-are-currently-busy%2C-waiting.-Increase-maxThreads-%28400%29-or-check-the-servlet-status-tp28150773p28150773.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status
On 06/04/2010 13:03, kapilok wrote: I need to restart tomcat after the error: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status This happens when the Database gets overloaded and responds slowly. I would like tomcat to recover (without having to restart), once I have solved the database overload issue (which takes a few minutes). I am using Tomcat 5.5.20 on Windows Server 2003. Please advise. Tomcat shouldn't need to restart in this case. It should recover automatically. If it doesn't, that suggests your applciation is not handling the lack of response from the database in a graceful manner. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark, On 4/6/2010 8:54 AM, Mark Thomas wrote: On 06/04/2010 13:03, kapilok wrote: I need to restart tomcat after the error: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status This happens when the Database gets overloaded and responds slowly. I would like tomcat to recover (without having to restart), once I have solved the database overload issue (which takes a few minutes). I am using Tomcat 5.5.20 on Windows Server 2003. Please advise. Tomcat shouldn't need to restart in this case. It should recover automatically. If it doesn't, that suggests your applciation is not handling the lack of response from the database in a graceful manner. +1 Sounds like deadlock on the application to me: this would tie-up threads forever (the definition of deadlock: duh). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku7SIQACgkQ9CaO5/Lv0PCNuwCgupCY3VQUDaVv2cecPeDbtvIr rXcAnjlRDztDldq3oBAzZwoLIvkcA+r7 =e/vU -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status
markt-2 wrote: On 06/04/2010 13:03, kapilok wrote: I need to restart tomcat after the error: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status This happens when the Database gets overloaded and responds slowly. I would like tomcat to recover (without having to restart), once I have solved the database overload issue (which takes a few minutes). I am using Tomcat 5.5.20 on Windows Server 2003. Please advise. Tomcat shouldn't need to restart in this case. It should recover automatically. If it doesn't, that suggests your applciation is not handling the lack of response from the database in a graceful manner. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Does Tomcat guarantee recovery in case of this Error? Can you point me to the documentation ? Please suggest a way to test / validate the recovery. Thanks, Kapil -- View this message in context: http://old.nabble.com/Recover-from-SEVERE%3A-All-threads-%28400%29-are-currently-busy%2C-waiting.-Increase-maxThreads-%28400%29-or-check-the-servlet-status-tp28150773p28153331.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kapilok, On 4/6/2010 11:19 AM, kapilok wrote: Does Tomcat guarantee recovery in case of this Error? The JVM/Tomcat should recover in the sense that service threads that complete their work should go back into the thread pool and accept new requests. Unfortunately, your webapp can significantly impact the ability for those threads to complete their request handling. Can you point me to the documentation? There is no documentation that covers this recovery: it's just a feature of using thread pools. Please suggest a way to test / validate the recovery. First, figure out what the real problem is. The symptom is obvious (see your subject line). To see what those threads are doing, allow this situation to occur, then do the following: 1. Perform whatever magic you need to solve the database overload issue 2. Wait a minute or two to give your webapp a chance to breathe 3. Perform a thread dump on your webapp (http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F) 4. Wait 30 seconds 5. Perform another thread dump on your webapp 6. Inspect the thread dump(s) to see what your threads are stuck doing. If they're stuck in your webapp's code, go fix that and try again. If they're stuck in Tomcat's code, post back and we'll take a look. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku7WkgACgkQ9CaO5/Lv0PA/rgCffQZFnykd/0eXKdbOxlY/6XA9 TskAoLNKqjjcJbrAyoKeqT5J6TRnsVKX =KvA6 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
