Christopher Schultz-2 wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Kapilok,
>
> On 4/6/2010 11:19 AM, kapilok wrote:
>> Does Tomcat guarantee recovery in case of this Error?
>
> The JVM/Tomcat should "recover" in the sense that service threads that
> complete their work should go back into the thread pool and accept new
> requests.
>
> Unfortunately, your webapp can significantly impact the ability for
> those threads to complete their request handling.
>
>> Can you point me to the documentation?
>
> There is no documentation that covers this "recovery": it's just a
> feature of using thread pools.
>
>> Please suggest a way to test / validate the recovery.
>
> First, figure out what the real problem is. The symptom is obvious (see
> your subject line).
>
> To see what those threads are doing, allow this situation to occur, then
> do the following:
>
> 1. Perform whatever magic you need to "solve" the "database overload"
> issue
>
> 2. Wait a minute or two to give your webapp a chance to breathe
>
> 3. Perform a thread dump on your webapp
> (http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F)
>
> 4. Wait 30 seconds
>
> 5. Perform another thread dump on your webapp
>
> 6. Inspect the thread dump(s) to see what your threads are stuck doing.
> If they're stuck in your webapp's code, go fix that and try again. If
> they're stuck in Tomcat's code, post back and we'll take a look.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAku7WkgACgkQ9CaO5/Lv0PA/rgCffQZFnykd/0eXKdbOxlY/6XA9
> TskAoLNKqjjcJbrAyoKeqT5J6TRnsVKX
> =KvA6
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>
Chris,
I think the thread dump does suggest that it is the webapp code (c3p0
connection pool) and not tomcat. But Please take a look anyways: (pasted
partial dump only)
Here's my test scenario:
1. Start Tomcat (with maxThreads="40" )
2. Run JMeter Load (40 concurrent with some ramp up)
- All requests succeed
3. Now load the database with some heavy process, so CPU consumption is high
4. Run same jMeter load; now response times are slow
5. Get ThreadPool Full Error; Take Thread Dump
6. Kill all JMeter requests; so Tomcat can breathe
7. Wait 10 minutes. Try login to webapp - Cant login. Browser does not
display Login page or any other page.
8. Take another Thread Dump. Its the same. NOTE: Main Thread and
TP-Processor4 are still running (observed with VisualVM1.1 on JDK 6)
9. Try running jMeter load. No reponse - No requests going through.
10. Waited 30 minutes and had no option but to bounce Tomcat.
Bottom Line: What can I do so I don't have to bounce Tomcat when I run into
this situation?
Partial Thread Dump
***
"http-9080-Processor40" daemon prio=6 tid=0x4d1ba000 nid=0x1e0 in
Object.wait() [0x5162e000..0x5162fd94]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x07b12648> (a
com.mchange.v2.resourcepool.BasicResourcePool)
at
com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at
com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
- locked <0x07b12648> (a com.mchange.v2.resourcepool.BasicResourcePool)
at
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:523)
at
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:587)
at
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:612)
at
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:644)
...
"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon
prio=6 tid=0x4dcda400 nid=0x6ac in Object.wait() [0x5068f000..0x5068fb94]
java.lang.Thread.State: TIM
