Here are a couple patches that you might find useful.
Patch 1: jakarta-commons/pool (implementation of TODO in
GenericKeyedObjectPool)
If maxTotal is specified for genericKeyedObjectPool, then clearOldest method
is called which removes the oldest 15% of the objects from the pool:
basically the cache now implements an LRU for pruning. At some point it
would be nice if the percentage was configurable. I will add to bugzilla as
an attachemnt if it will let me.
Patch 2: jakarta-commons/dbcp
Currently there is not much configurability for the prepared statement
pooling. This patch defines a new property on the driveradaptercpds class:
maxPreparedStatements. If this value is set to something other than -1,
then the underlying statement pool will not use an evictor thread and will
use the mechanism in patch 1 to remove the oldest objects. If set to -1,
then the old behavior is used. You may find this patch useful. In our
environment we are concerned about creating a separate thread for EVERY
connection.
ToddC
-----Original Message-----
From: Dirk Verbeeck [mailto:[EMAIL PROTECTED]
Sent: Tuesday, February 24, 2004 1:51 PM
To: Todd Carmichael
Subject: Re: Prepared Statement pooling
Todd
The code looks just fine, building a TreeMap is needed if you want to
remove multiple items at once. A small optimalization would be to
limit the number of items you put in the map.
(if the size>itemsToRemove remove the youngest item)
But I don't see a way to avoid the sorted map.
If you can make some JUnit tests then I will commit this weekend.
Cheers
Dirk
Todd Carmichael wrote:
> Dirk,
> Here is the code that will clear the oldest items in a pool (basically
> an LRU). I would appreciate any thoughts you have concerning this
> code. It removes the oldest 25% of the items in the cache. The
> number of items to remove could definitely be a configurable option.
> Is building the TreeMap (sorting the items) necessary (i.e. is there a
> better way to determine the oldest items)?
>
> Thanks.
>
> class ObjectTimeStampComparable implements Comparator
> {
> public int compare(Object o1, Object o2)
> {
> return (int)(((ObjectTimestampPair)o1).tstamp -
> ((ObjectTimestampPair)o2).tstamp);
> }
> }
> public synchronized void clearOldest() {
>
> TreeMap map = new TreeMap(new ObjectTimeStampComparable());
> for(Iterator keyiter = _poolList.iterator(); keyiter.hasNext(); )
{
> Object key = keyiter.next();
> CursorableLinkedList list =
> (CursorableLinkedList)(_poolMap.get(key));
> for(Iterator it = list.iterator(); it.hasNext(); ) {
> try {
> // each item into the map uses the
> objectimestamppair object
> // as the key. It then gets sorted based on the
> timstamp field
> // each value in the map is the parent list it
> belongs in.
> ObjectTimestampPair pair =
> (ObjectTimestampPair)(it.next());
> map.put(pair,key);
> } catch(Exception e) {
> // ignore error, keep destroying the rest
> }
> }
> }
> Set setPairKeys = map.entrySet();
> int itemsToRemove = (int)(_maxTotal*.25) + 1; // 25% of items
> removed plus one to account for zero
> Iterator iter = setPairKeys.iterator();
> while(iter.hasNext() && itemsToRemove>0)
> {
> Map.Entry entry = (Map.Entry)iter.next();
> // kind of backwards on naming. In the map, each key is
> the objecttimestamppair
> // because it has the ordering with the timestamp value.
> Each value that the
> // key references is the key of the list it belongs to.
> CursorableLinkedList list =
> (CursorableLinkedList)(_poolMap.get(entry.getValue()));
> list.remove(entry.getKey());
> try {
> _factory.destroyObject(entry.getValue(),entry.getKey());
> }
> catch(Exception e) {
> // ignore error, keep destroying the rest
> }
> _totalIdle--;
> itemsToRemove--;
> }
> notifyAll();
> }
>
>
> ToddC
Index: src/java/org/apache/commons/pool/impl/GenericKeyedObjectPool.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/pool/src/java/org/apache/commons/pool/impl/GenericKeyedObjectPool.java,v
retrieving revision 1.26
diff -u -r1.26 GenericKeyedObjectPool.java
--- src/java/org/apache/commons/pool/impl/GenericKeyedObjectPool.java 28 Feb 2004
11:46:33 -0000 1.26
+++ src/java/org/apache/commons/pool/impl/GenericKeyedObjectPool.java 29 Feb 2004
00:27:26 -0000
@@ -19,6 +19,10 @@
import java.util.HashMap;
import java.util.Iterator;
import java.util.NoSuchElementException;
+import java.util.TreeMap;
+import java.util.Comparator;
+import java.util.Set;
+import java.util.Map;
import org.apache.commons.collections.CursorableLinkedList;
import org.apache.commons.pool.BaseKeyedObjectPool;
@@ -733,10 +737,9 @@
// otherwise
if(null == pair) {
// if there is a totalMaxActive and we are at the limit then
- // we have to make room
- // TODO: this could be improved by only removing the oldest object
+ // we have to make room.
if ((_maxTotal > 0) && (_totalActive + _totalIdle >= _maxTotal)) {
- clear();
+ clearOldest();
}
// check if we can create one
@@ -807,6 +810,84 @@
_totalIdle = 0;
notifyAll();
}
+
+ /**
+ * implementation of comparator interface for ObjectTimeStampPair objects
+ * used in sorting all idle objects in pool so that the oldest ones can
+ * be removed
+ *
+ */
+ class ObjectTimeStampComparable implements Comparator
+ {
+ public int compare(Object o1, Object o2)
+ {
+ long iDiff = ((ObjectTimestampPair)o1).tstamp -
((ObjectTimestampPair)o2).tstamp;
+ if (iDiff==0)
+ {
+ iDiff = o1.hashCode() - o2.hashCode();
+ }
+ return (int)iDiff;
+ }
+ }
+
+ /**
+ *
+ * Method clears oldest 15% of objects in pool. The method sorts the
+ * objects into a TreeMap and then iterates the first 15% for removal
+ *
+ *
+ */
+ public synchronized void clearOldest() {
+
+ // build sorted map of idle objects
+ TreeMap map = new TreeMap(new ObjectTimeStampComparable());
+ for(Iterator keyiter = _poolList.iterator(); keyiter.hasNext(); ) {
+ Object key = keyiter.next();
+ CursorableLinkedList list = (CursorableLinkedList)(_poolMap.get(key));
+ for(Iterator it = list.iterator(); it.hasNext(); ) {
+ try {
+ // each item into the map uses the objectimestamppair object
+ // as the key. It then gets sorted based on the timstamp field
+ // each value in the map is the parent list it belongs in.
+ ObjectTimestampPair pair = (ObjectTimestampPair)(it.next());
+ map.put(pair,key);
+ } catch(Exception e) {
+ // ignore error, keep destroying the rest
+ }
+ }
+ }
+ // now iterate created map and kill the first 15%. Should be a
+ // configurable option.
+ Set setPairKeys = map.entrySet();
+ int itemsToRemove = ((int)(map.size()*.15)) + 1; // 15% of items removed plus
one to account for zero
+ Iterator iter = setPairKeys.iterator();
+ while(iter.hasNext() && itemsToRemove>0)
+ {
+ Map.Entry entry = (Map.Entry)iter.next();
+ // kind of backwards on naming. In the map, each key is the
objecttimestamppair
+ // because it has the ordering with the timestamp value. Each value that
the
+ // key references is the key of the list it belongs to.
+ Object key = entry.getValue();
+ ObjectTimestampPair pairTimeStamp = (ObjectTimestampPair) entry.getKey();
+ CursorableLinkedList list = (CursorableLinkedList)(_poolMap.get(key));
+ list.remove(pairTimeStamp);
+
+ try {
+ _factory.destroyObject(key,pairTimeStamp.value);
+ }
+ catch(Exception e) {
+ // ignore error, keep destroying the rest
+ }
+ // if that was the last object for that key, drop that pool
+ if (list.isEmpty()) {
+ _poolMap.remove(key);
+ _poolList.remove(key);
+ }
+ _totalIdle--;
+ itemsToRemove--;
+ }
+ notifyAll();
+ }
public synchronized void clear(Object key) {
CursorableLinkedList pool = (CursorableLinkedList)(_poolMap.remove(key));
Index: src/test/org/apache/commons/pool/impl/TestGenericKeyedObjectPool.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/pool/src/test/org/apache/commons/pool/impl/TestGenericKeyedObjectPool.java,v
retrieving revision 1.18
diff -u -r1.18 TestGenericKeyedObjectPool.java
--- src/test/org/apache/commons/pool/impl/TestGenericKeyedObjectPool.java 28 Feb
2004 11:46:11 -0000 1.18
+++ src/test/org/apache/commons/pool/impl/TestGenericKeyedObjectPool.java 29 Feb
2004 00:27:55 -0000
@@ -221,6 +221,51 @@
assertEquals(0, pool.getNumIdle("b"));
}
+ public void testMaxTotalLRU() throws Exception {
+ pool.setMaxActive(2);
+ pool.setMaxTotal(3);
+ pool.setWhenExhaustedAction(GenericKeyedObjectPool.WHEN_EXHAUSTED_GROW);
+
+ Object o1 = pool.borrowObject("a");
+ assertNotNull(o1);
+ long lo1Hash = o1.hashCode();
+ pool.returnObject("a", o1);
+ Thread.sleep(10);
+
+ Object o2 = pool.borrowObject("b");
+ assertNotNull(o2);
+ pool.returnObject("b", o2);
+ Thread.sleep(10);
+
+ Object o3 = pool.borrowObject("c");
+ assertNotNull(o3);
+ pool.returnObject("c", o3);
+ Thread.sleep(10);
+
+ Object o6 = pool.borrowObject("a");
+ assertNotNull(o6);
+ long lo6Hash = o6.hashCode();
+ pool.returnObject("a", o6);
+ Thread.sleep(10);
+
+ assertTrue(lo1Hash == lo6Hash);
+
+ // this should cause a to be bumped out of the pool
+ Object o4 = pool.borrowObject("d");
+ assertNotNull(o4);
+ pool.returnObject("d", o4);
+ Thread.sleep(10);
+
+ // now re-request b, we should get a different object because it should
+ // have been expelled from pool
+ Object o5 = pool.borrowObject("b");
+ assertNotNull(o5);
+ long lo5Hash = o5.hashCode();
+ pool.returnObject("b", o5);
+ assertTrue(lo5Hash != lo1Hash);
+
+ }
+
public void testSettersAndGetters() throws Exception {
GenericKeyedObjectPool pool = new GenericKeyedObjectPool();
Index: src/java/org/apache/commons/dbcp/cpdsadapter/DriverAdapterCPDS.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/dbcp/src/java/org/apache/commons/dbcp/cpdsadapter/DriverAdapterCPDS.java,v
retrieving revision 1.8
diff -u -r1.8 DriverAdapterCPDS.java
--- src/java/org/apache/commons/dbcp/cpdsadapter/DriverAdapterCPDS.java 28 Feb 2004
12:18:17 -0000 1.8
+++ src/java/org/apache/commons/dbcp/cpdsadapter/DriverAdapterCPDS.java 29 Feb 2004
00:26:10 -0000
@@ -113,6 +113,7 @@
private int _timeBetweenEvictionRunsMillis = -1;
private int _numTestsPerEvictionRun = -1;
private int _minEvictableIdleTimeMillis = -1;
+ private int _maxPreparedStatements = -1;
private boolean getConnectionCalled = false;
@@ -147,13 +148,28 @@
*/
KeyedObjectPool stmtPool = null;
if (isPoolPreparedStatements()) {
- stmtPool = new GenericKeyedObjectPool(null,
- getMaxActive(), GenericKeyedObjectPool.WHEN_EXHAUSTED_GROW, 0,
- getMaxIdle(), false, false, getTimeBetweenEvictionRunsMillis(),
- getNumTestsPerEvictionRun(),
- getMinEvictableIdleTimeMillis(), false);
+ if (getMaxPreparedStatements() == -1)
+ {
+ // since there is no limit, create a prepared statement pool with an
eviction thread
+ // evictor settings are the same as the connection pool settings.
+ stmtPool = new GenericKeyedObjectPool(null,
+ getMaxActive(), GenericKeyedObjectPool.WHEN_EXHAUSTED_GROW, 0,
+ getMaxIdle(), false, false,
+
getTimeBetweenEvictionRunsMillis(),getNumTestsPerEvictionRun(),getMinEvictableIdleTimeMillis(),
+ false);
+ }
+ else
+ {
+ // since there is limit, create a prepared statement pool without an
eviction thread
+ // pool has LRU functionality so when the limit is reached, 15% of
the pool is cleared.
+ // see
org.apache.commons.pool.impl.GenericKeyedObjectPool.clearOldest method
+ stmtPool = new GenericKeyedObjectPool(null,
+ getMaxActive(), GenericKeyedObjectPool.WHEN_EXHAUSTED_GROW, 0,
+ getMaxIdle(), getMaxPreparedStatements(), false, false,
+ -1,0,0, // -1 tells the pool that there should be no eviction
thread.
+ false);
+ }
}
-
// Workaround for buggy WebLogic 5.1 classloader - ignore the
// exception upon first invocation.
try {
@@ -201,6 +217,8 @@
String.valueOf(getNumTestsPerEvictionRun())));
ref.add(new StringRefAddr("minEvictableIdleTimeMillis",
String.valueOf(getMinEvictableIdleTimeMillis())));
+ ref.add(new StringRefAddr("maxPreparedStatements",
+ String.valueOf(getMaxPreparedStatements())));
return ref;
}
@@ -275,6 +293,11 @@
setMinEvictableIdleTimeMillis(
Integer.parseInt(ra.getContent().toString()));
}
+ ra = ref.get("maxPreparedStatements");
+ if (ra != null && ra.getContent() != null) {
+ setMaxPreparedStatements(
+ Integer.parseInt(ra.getContent().toString()));
+ }
cpds = this;
}
@@ -552,5 +575,14 @@
public void setMinEvictableIdleTimeMillis(int minEvictableIdleTimeMillis) {
assertInitializationAllowed();
_minEvictableIdleTimeMillis = minEvictableIdleTimeMillis;
+ }
+ public int getMaxPreparedStatements()
+ {
+ return _maxPreparedStatements;
+ }
+
+ public void setMaxPreparedStatements(int maxPreparedStatements)
+ {
+ _maxPreparedStatements = maxPreparedStatements;
}
}
Index: src/test/org/apache/commons/dbcp/datasources/TestSharedPoolDataSource.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/dbcp/src/test/org/apache/commons/dbcp/datasources/TestSharedPoolDataSource.java,v
retrieving revision 1.7
diff -u -r1.7 TestSharedPoolDataSource.java
--- src/test/org/apache/commons/dbcp/datasources/TestSharedPoolDataSource.java 28 Feb
2004 11:47:52 -0000 1.7
+++ src/test/org/apache/commons/dbcp/datasources/TestSharedPoolDataSource.java 29 Feb
2004 00:26:55 -0000
@@ -412,7 +412,6 @@
conn3.close();
}
-
// Bugzilla Bug 24136 ClassCastException in DriverAdapterCPDS
// when setPoolPreparedStatements(true)
public void testPoolPrepareStatement() throws Exception
@@ -429,5 +428,80 @@
rset.close();
stmt.close();
conn.close();
+ }
+
+ public void testPoolPreparedStatements() throws Exception {
+ DriverAdapterCPDS mypcds = new DriverAdapterCPDS();
+ DataSource myds = null;
+ mypcds.setDriver("org.apache.commons.dbcp.TesterDriver");
+ mypcds.setUrl("jdbc:apache:commons:testdriver");
+ mypcds.setUser("foo");
+ mypcds.setPassword("bar");
+ mypcds.setPoolPreparedStatements(true);
+ mypcds.setMaxPreparedStatements(10);
+
+ SharedPoolDataSource tds = new SharedPoolDataSource();
+ tds.setConnectionPoolDataSource(mypcds);
+ tds.setMaxActive(getMaxActive());
+ tds.setMaxWait((int)(getMaxWait()));
+ tds.setDefaultTransactionIsolation(
+ Connection.TRANSACTION_READ_COMMITTED);
+
+ myds = tds;
+
+ Connection conn = ds.getConnection();
+ PreparedStatement stmt = null;
+ ResultSet rset = null;
+
+ assertTrue(null != conn);
+
+ stmt = conn.prepareStatement("select * from dual");
+ assertTrue(null != stmt);
+ long l1HashCode = stmt.hashCode();
+ rset = stmt.executeQuery();
+ assertTrue(null != rset);
+ assertTrue(rset.next());
+ rset.close();
+ stmt.close();
+
+ stmt = conn.prepareStatement("select * from dual");
+ assertTrue(null != stmt);
+ long l2HashCode = stmt.hashCode();
+ rset = stmt.executeQuery();
+ assertTrue(null != rset);
+ assertTrue(rset.next());
+ rset.close();
+ stmt.close();
+
+ // statement pooling is not enabled, we should get different statements
+ assertTrue(l1HashCode != l2HashCode);
+ conn.close();
+ conn = null;
+
+ conn = myds.getConnection();
+
+ stmt = conn.prepareStatement("select * from dual");
+ assertTrue(null != stmt);
+ long l3HashCode = stmt.hashCode();
+ rset = stmt.executeQuery();
+ assertTrue(null != rset);
+ assertTrue(rset.next());
+ rset.close();
+ stmt.close();
+
+ stmt = conn.prepareStatement("select * from dual");
+ assertTrue(null != stmt);
+ long l4HashCode = stmt.hashCode();
+ rset = stmt.executeQuery();
+ assertTrue(null != rset);
+ assertTrue(rset.next());
+ rset.close();
+ stmt.close();
+
+ // prepared statement pooling is working
+ assertTrue(l3HashCode == l4HashCode);
+ conn.close();
+ conn = null;
+
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
