Post your datasource resource definition. You may miss something like
<parameter>
<name>testOnBorrow</name>
<value>true</value>
</parameter>
<parameter>
<name>testOnReturn</name>
<value>true</value>
</parameter>
<parameter>
<name>minEvictableIdleTimeMillis</name>
<value>-1</value>
</parameter>
<parameter>
<name>timeBetweenEvictionRunsMillis</name>
<value>300000</value>
</parameter>
<parameter>
<name>numTestsPerEvictionRun</name>
<value>1</value>
</parameter>
<parameter>
<name>testWhileIdle</name>
<value>true</value>
</parameter>
-----Original Message-----
From: Kaleb Pederson [mailto:[EMAIL PROTECTED]
Sent: October 4, 2004 11:19 AM
To: Tomcat Users List
Subject: Re: Tomcat becoming almost completely non-responsive
John,
Thanks for the feedback. My code is actually quite simple and I have
everything factored out to single function that handles the DB code. I'll
paste it below for you, and others to look at. *Please let me know* if
there
is an exit path that I somehow missed.
I already have tomcat setup to log abandoned connections. At your
instigation, I grep'd the logs and found a few instances of it in the logs
which I was unaware of. Each one of them corresponds to a time we had high
load on the server, so I, hopefully not niavely ;), am guesssing that the
abandoned timeout was shorter (currently 60 seconds) than the time it took
the server to respond to the request (up to 5 minutes). I guess I should
up
the abandoned timeout as that could flame the problem.
If I'm interpreting the thread dump and mysqladmin processlist information
correctly, what I'm seeing is that MySQL considers the DB connection to be
sleeping, and that 48 of my active db threads were in object.wait() after a
call to AbandonedObjectPool.borrowObject() which seems to get called in
every case after a call to DataSource.getConnection():
// note that I removed the org.apache.commons prefix
48 like "TP-Processor312" daemon prio=1 tid=0x70331680 nid=0x5e87 waiting
for
monitor entry [6df79000..6df7a87c] at
dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.jav
a
:316)
- waiting to lock <0x7b071438> (a dbcp.PoolableConnectionFactory)
at pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:833)
at dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:119)
at dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
at dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
// the function in my filter that checks the DB.
at RequestFilterServlet.getUrlMapping(RequestFilterServlet.java:187)
...
Here's the code that handles the db:
protected RedirectInfo getUrlMapping(String requestURI) {
RedirectInfo retval = null;
Connection conn = null;
Statement statement = null;
ResultSet result = null;
try {
conn = ds.getConnection();
statement = conn.createStatement();
statement.setEscapeProcessing(true);
result = statement.executeQuery(QUERY_UNIQUE_MATCH + quote(requestURI));
if (!result.first()) {
result = statement.executeQuery(QUERY_CHILD_MATCH_1 + quote(requestURI)
+
QUERY_CHILD_MATCH_2);
}
// get the information
if (result.first()) {
retval = new RedirectInfo();
retval.setURL(result.getString(1));
retval.setCode(result.getInt(2));
debug(requestURI + " -> " + retval.getURL() + " [" +
Integer.toString(retval.getCode()) + "]");
} else {
debug("No matching redirect URL found for " + requestURI);
}
} catch(SQLException e) {
log("SQLException caught: " + e.getMessage(), e);
} finally {
try {
// cleanup after myself
if (statement != null) statement.close();
if (conn != null) conn.close();
} catch (SQLException e) {
log("Error cleaning up connection/statement: " + e.getMessage(),e);
}
}
return retval;
}
Thanks for all the help.
--Kaleb
On Sunday 03 October 2004 1:58 pm, [EMAIL PROTECTED] wrote:
The most likely reason for your connections not being re-used is that
your java code isn't explicitly closing connections, statements and
result sets after use. If visiting all your code is too onerous a
task, you could add the following parameters to your data-source
configuration:
<parameter>
<name>removeAbandoned</name>
<value>true</value>
</parameter>
<parameter>
<name>removeAbandonedTimeout</name>
<value>60</value>
</parameter>
The effect of the above is to mark any data-pool connection as
re-usable after 60 seconds of inactivity. You can add the following to
log abandonned connection activity:
<parameter>
<name>logAbandoned</name>
<value>true</value>
</parameter>
John Thompson
|---------+---------------------------->
|
| | Kaleb Pederson |
| | <[EMAIL PROTECTED]|
| | wu.edu> |
| |
| | 02/10/2004 10:07 |
| | AM |
| | Please respond to|
| | "Tomcat Users |
| | List" |
|
|---------+---------------------------->
------------------------------------------------------------------------
>--------------------------------------|
>
| To: Tomcat Users List <[EMAIL PROTECTED]>
|
| | cc:
|
| Subject: Tomcat becoming almost completely non-responsive
------------------------------------------------------------------------
>--------------------------------------|
Hello,
I have an interesting problem. After a while, tomcat (5.0.27) becomes
almost completely non-responsive. If I telnet in to port 8009 (I'm
using apache and mod_jk2), I get no response, at least not within the
default timeout. If
a browse to a page, I will generally, after about 4-5 minutes, see a page
returned.
To narrow down the slowness, I generated a full thread dump, and found
the following information:
[ see attachment for more info]
Total threads: 180
executeQuery: 2 // executing a db query
validateConnection: 0 // trying to validate their connection
validateObect: 48 // in
commons.dbcp.PoolableConnectionFactory.validateObject
socketAccept: 3 // accepting a socket
socketRead0: 10 // reading a socket
ReferenceQueue: 1
ThreadPool$MonitorRunnable: 2
borrowObject and Object.wait: 85 // trying to get an object from the
pool
Object.wait: 20 // threads just waiting around
Remaining: 9 // misc. threads
My database connection is setup so that I have 50 allowed connections,
which matches my 48 in validateObject and 2 executing queries.
However, when I query the database status, I see 2 active threads and
the rest are 'sleeping', just waiting around, as they would be if the
connection pool hadn't released them yet.
So, why would there be 48 connections that seemed locked and weren't
querying the DB? And then the other 85 that were seemingly waiting on
the 45? Any ideas what might be going on? The DB is ready? I have
log abandoned turned
and an haven't seen a problem yet. If the load drops sufficiently on the
server, everything eventually returns back to normal, otherwise it takes
5-10
minutes to get a response from the server.
I have attached an abbreviated form of the thread dump which should
provide all the critical information and can provide as much other
information as is necessary.
Thanks for the help. *All* suggestions welcome ;)
--Kaleb
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
!DSPAM:41616a0b202643974811692!
