[
https://issues.apache.org/jira/browse/DBCP-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816685#comment-17816685
]
Phil Steitz commented on DBCP-595:
----------------------------------
Thanks for the info on validation. That info and the partial thread dump above
(can you maybe include a fuller trace?) makes it look likely to me that the
problem is that that the application is not closing connections on some
exception paths. Can you check to make sure that even on exception paths, the
code is calling close on the connection handles returned by DBCP? The fact
that forcing close on fatal exceptions improves things supports the hypothesis
that what is going on is the client code is not closing connections on some
exception paths.
Also, can you confirm that when the starvation happens, the pool is reporting
numActive == maxActive?
With validation turned on, DBCP will reliably close connections when validation
fails. If any kind of exception occurs during the close, DBCP will destroy the
connection handle and reduce its active count. You can see that in the
validateObject method of PoolableConnectionFactory. If you set
fastFailValidation to true, DBCP will fail validation without attempting a
validation query after a fatal exception occurs, again destroying the handle
and adjusting counters.
> Connection pool can be exhausted when connections are killed on the DB side
> ---------------------------------------------------------------------------
>
> Key: DBCP-595
> URL: https://issues.apache.org/jira/browse/DBCP-595
> Project: Commons DBCP
> Issue Type: Bug
> Affects Versions: 2.11.0
> Reporter: Dénes Bodó
> Priority: Critical
> Labels: deadlock, robustness
>
> Apache Oozie 5.2.1 uses OpenJPA 2.4.2 and commons-dbcp 1.4 and commons-pool
> 1.5.4. These are ancient versions, I know.
> h1. Description
> The issue is that when due to some network issues or "maintenance work" on
> the DB side (especially PostgreSQL) which causes the DB connection to be
> closed, it results exhausted Pool on the client side. Many threads are
> waiting at this point:
> {noformat}
> "pool-2-thread-4" #20 prio=5 os_prio=31 tid=0x00007faf7903b800 nid=0x8603
> waiting on condition [0x000000030f3e7000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000066aca8e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:1324)
> {noformat}
> According to my observation this is because the JDBC driver does not get
> closed on the client side, nor the abstract DBCP connection
> _org.apache.commons.dbcp2.PoolableConnection_ .
> h1. Repro
> (Un)Fortunately I can reproduce the issue using the latest and greatest
> commons-dbcp 2.11.0 and commons-pool 2.12.0 along with OpenJPA 3.2.2.
> I've just created a Java application to reproduce the issue:
> [https://github.com/dionusos/pool_exhausted_repro] . See README.md for
> detailed repro steps.
> h1. Kind of solution?
> To be honest I am not really familiar with DBCP but with this change I
> managed to make my application more robust:
> {code:java}
> diff --git a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> index 440cb756..678550bf 100644
> --- a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> +++ b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> @@ -214,6 +214,10 @@ public class PoolableConnection extends
> DelegatingConnection<Connection> impleme
> @Override
> protected void handleException(final SQLException e) throws SQLException
> {
> fatalSqlExceptionThrown |= isFatalException(e);
> + if (fatalSqlExceptionThrown && getDelegate() != null) {
> + getDelegate().close();
> + this.close();
> + }
> super.handleException(e);
> }{code}
> What do you think about this approach?
> Is it a completely dead-end or we can start working on it in this direction?
> Do you agree that the reported and reproduced issue is a real one and nut
> just some kind of misconfiguration?
>
> I am lost at this point and I need to move forward so I am asking for
> guidance here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
