Hello, I'm not a developer on the project I'm supporting, but I have
repeatedly seen this happen and would appreciate some input or advice.
We're using version 1.6 of the commons pool, I don't believe we could
upgrade without good reason.
We're running WebLogic and periodically see thread count shoot up to
the work manager maximum, and throughput grinds to a halt. All threads
are blocked waiting on the commons pool, which I thought was strange
as heap dumps show output like:
Type Name Value
int evictLastIndex -1
ref _evictor null
int _numActive 206
ref _factory
org.springframework.aop.target.CommonsPoolTargetSource @ 0x610...
ref _pool java.util.LinkedList @ 0x610...
long _softMinEvictableIdleTimeMillis-1
long _minEvictableIdleTimeMillis 1800000
int _numTestsPerEvictionRun 3
long _timeBetweenEvictionRunsMillis -1
boolean_testWhileIdle FALSE
boolean_testOnReturn FALSE
boolean_testOnBorrow FALSE
byte _whenExhaustedAction 1
long _maxWait -1
int _maxActive 4096
int _minIdle 10
int _maxIdle 8
booleanclosed FALSE
So from that, I would have expected numActive to be much closer to
4096 which is the maxActive we set. Am I looking at the wrong place?
Why is this pool blocking? This has happened multiple times, and each
time numActive is between 200 and 300. The pool is being used (I believe)
to limit the number of concurrent security authorizations, so each borrow
then pings an external service, and that external service does not appear
overloaded at all.
I did notice that when I drilled down into the pool's linked list, it
only ever has 8 members in it, I would have expected numActive members
at least. I don't fully understand the pool though.
Here's a partial thread dump showing this behavior (addresses truncated,
but they all blocked on the same pool object):
"[ACTIVE] ExecuteThread: '160' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1f
waiting for monitor entry [0x000...]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.pool.impl.GenericObjectPool.returnObject(
GenericObjectPool.java:916)
- waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
at org.springframework.aop.target.CommonsPoolTargetSource.
releaseTarget(CommonsPoolTargetSource.java:252)
--
"[ACTIVE] ExecuteThread: '159' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x0000... nid=0x4b1e
waiting for monitor entry [0x000...]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(
GenericObjectPool.java:781)
- waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
at org.springframework.aop.target.CommonsPoolTargetSource.getTarget(
CommonsPoolTargetSource.java:244)
--
"[ACTIVE] ExecuteThread: '158' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1d
waiting for monitor entry [0x000...]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.pool.impl.GenericObjectPool.returnObject(
GenericObjectPool.java:916)
- waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
at org.springframework.aop.target.CommonsPoolTargetSource.
releaseTarget(CommonsPoolTargetSource.java:252)
--
"[ACTIVE] ExecuteThread: '157' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1c
waiting for monitor entry [0x000...]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.pool.impl.GenericObjectPool.returnObject(
GenericObjectPool.java:916)
- waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
at org.springframework.aop.target.CommonsPoolTargetSource.
releaseTarget(CommonsPoolTargetSource.java:252)
--
"[ACTIVE] ExecuteThread: '156' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1b
waiting for monitor entry [0x000...]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.pool.impl.GenericObjectPool.returnObject(
GenericObjectPool.java:916)
- waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
at org.springframework.aop.target.CommonsPoolTargetSource.
releaseTarget(CommonsPoolTargetSource.java:252)
--
"[ACTIVE] ExecuteThread: '155' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1a
waiting for monitor entry [0x000...]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(
GenericObjectPool.java:781)
- waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
at org.springframework.aop.target.CommonsPoolTargetSource.getTarget(
CommonsPoolTargetSource.java:244)
--
... repeated hundreds of times.
Thanks for any advice you may have.
