On 4/13/15 9:36 AM, Dan wrote: > Hello, I'm not a developer on the project I'm supporting, but I have > repeatedly seen this happen and would appreciate some input or advice. > We're using version 1.6 of the commons pool, I don't believe we could > upgrade without good reason. From the line numbers in the stack trace, it looks like you are actually running pool 1.3, which is, well, ancient. You should verify the version and if it is as I suspect, you should definitely upgrade. Just have a look at the change log for the many, many issues that have been resolved since 1.3. That version should be deadlock-free, though it achieves that by extreme over-synchronization. Both borrow and return are fully synchronized (threads are waiting on the pool monitor in the dump below). Version 1.3 is the least performant version of commons pool.
Additionally, the maxIdle setting limits the number of instances that can be idle in the pool to just 8. That means that when instances are returned and there are already 8 idle, the returning instances are destroyed. When more load arrives, you then have to wait for them to be created, which in v 1.3 causes all threads to wait on the factory. If you can afford to have more instances idle, you should increase that number. You will likely get immediate relief by increasing maxIdle to several hundred or even the maxActive number; but you really should upgrade to a more recent version. See the pool web page for JDK requirements and version compatibility. Phil > > We're running WebLogic and periodically see thread count shoot up to > the work manager maximum, and throughput grinds to a halt. All threads > are blocked waiting on the commons pool, which I thought was strange > as heap dumps show output like: > > Type Name Value > int evictLastIndex -1 > ref _evictor null > int _numActive 206 > ref _factory > org.springframework.aop.target.CommonsPoolTargetSource @ 0x610... > ref _pool java.util.LinkedList @ 0x610... > long _softMinEvictableIdleTimeMillis-1 > long _minEvictableIdleTimeMillis 1800000 > int _numTestsPerEvictionRun 3 > long _timeBetweenEvictionRunsMillis -1 > boolean_testWhileIdle FALSE > boolean_testOnReturn FALSE > boolean_testOnBorrow FALSE > byte _whenExhaustedAction 1 > long _maxWait -1 > int _maxActive 4096 > int _minIdle 10 > int _maxIdle 8 > booleanclosed FALSE > > So from that, I would have expected numActive to be much closer to > 4096 which is the maxActive we set. Am I looking at the wrong place? > Why is this pool blocking? This has happened multiple times, and each > time numActive is between 200 and 300. The pool is being used (I believe) > to limit the number of concurrent security authorizations, so each borrow > then pings an external service, and that external service does not appear > overloaded at all. > > I did notice that when I drilled down into the pool's linked list, it > only ever has 8 members in it, I would have expected numActive members > at least. I don't fully understand the pool though. > > Here's a partial thread dump showing this behavior (addresses truncated, > but they all blocked on the same pool object): > > "[ACTIVE] ExecuteThread: '160' for queue: 'weblogic.kernel.Default > (self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1f > waiting for monitor entry [0x000...] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.commons.pool.impl.GenericObjectPool.returnObject( > GenericObjectPool.java:916) > - waiting to lock <0x000...> (a > org.apache.commons.pool.impl.GenericObjectPool) > at org.springframework.aop.target.CommonsPoolTargetSource. > releaseTarget(CommonsPoolTargetSource.java:252) > -- > "[ACTIVE] ExecuteThread: '159' for queue: 'weblogic.kernel.Default > (self-tuning)'" daemon prio=10 tid=0x0000... nid=0x4b1e > waiting for monitor entry [0x000...] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.commons.pool.impl.GenericObjectPool.borrowObject( > GenericObjectPool.java:781) > - waiting to lock <0x000...> (a > org.apache.commons.pool.impl.GenericObjectPool) > at org.springframework.aop.target.CommonsPoolTargetSource.getTarget( > CommonsPoolTargetSource.java:244) > -- > "[ACTIVE] ExecuteThread: '158' for queue: 'weblogic.kernel.Default > (self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1d > waiting for monitor entry [0x000...] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.commons.pool.impl.GenericObjectPool.returnObject( > GenericObjectPool.java:916) > - waiting to lock <0x000...> (a > org.apache.commons.pool.impl.GenericObjectPool) > at org.springframework.aop.target.CommonsPoolTargetSource. > releaseTarget(CommonsPoolTargetSource.java:252) > -- > "[ACTIVE] ExecuteThread: '157' for queue: 'weblogic.kernel.Default > (self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1c > waiting for monitor entry [0x000...] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.commons.pool.impl.GenericObjectPool.returnObject( > GenericObjectPool.java:916) > - waiting to lock <0x000...> (a > org.apache.commons.pool.impl.GenericObjectPool) > at org.springframework.aop.target.CommonsPoolTargetSource. > releaseTarget(CommonsPoolTargetSource.java:252) > -- > "[ACTIVE] ExecuteThread: '156' for queue: 'weblogic.kernel.Default > (self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1b > waiting for monitor entry [0x000...] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.commons.pool.impl.GenericObjectPool.returnObject( > GenericObjectPool.java:916) > - waiting to lock <0x000...> (a > org.apache.commons.pool.impl.GenericObjectPool) > at org.springframework.aop.target.CommonsPoolTargetSource. > releaseTarget(CommonsPoolTargetSource.java:252) > -- > "[ACTIVE] ExecuteThread: '155' for queue: 'weblogic.kernel.Default > (self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1a > waiting for monitor entry [0x000...] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.commons.pool.impl.GenericObjectPool.borrowObject( > GenericObjectPool.java:781) > - waiting to lock <0x000...> (a > org.apache.commons.pool.impl.GenericObjectPool) > at org.springframework.aop.target.CommonsPoolTargetSource.getTarget( > CommonsPoolTargetSource.java:244) > -- > ... repeated hundreds of times. > > Thanks for any advice you may have. > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
