[jira] [Closed] (DBCP-595) Connection pool can be exhausted when connections are killed on the DB side
[
https://issues.apache.org/jira/browse/DBCP-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed DBCP-595.
Resolution: Not A Problem
Don't mean to be blunt / dismissive with the close here. But this does not
look like a DBCP bug and it would be better to discuss on the mailing list any
changes to behavior. See [~ggregory] 's nice summary there on why we do not
force close connections on fatal SQL exceptions - basically drivers don't
consistently return codes that a generic library can count on.
I would definitely recommend reporting or trying to find and patch the code
that checks out connections from the pool and does not close them on fatal
exception paths. That would impact a lot of other users. One way to find this
is to turn abandoned connection logging.
> Connection pool can be exhausted when connections are killed on the DB side
> ---
>
> Key: DBCP-595
> URL: https://issues.apache.org/jira/browse/DBCP-595
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.11.0
>Reporter: Dénes Bodó
>Priority: Critical
> Labels: deadlock, robustness
> Attachments: ReproOneThread-jstack-minIdle.txt,
> ReproOneThread-jstack_when_create.txt, ReproOneThread-jstack_when_stuck.txt,
> ReproOneThread-screenshot_when_create-newCreateCount_gt_localMaxTotal.png
>
>
> Apache Oozie 5.2.1 uses OpenJPA 2.4.2 and commons-dbcp 1.4 and commons-pool
> 1.5.4. These are ancient versions, I know.
> h1. Description
> The issue is that when due to some network issues or "maintenance work" on
> the DB side (especially PostgreSQL) which causes the DB connection to be
> closed, it results exhausted Pool on the client side. Many threads are
> waiting at this point:
> {noformat}
> "pool-2-thread-4" #20 prio=5 os_prio=31 tid=0x7faf7903b800 nid=0x8603
> waiting on condition [0x00030f3e7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066aca8e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:1324)
> {noformat}
> According to my observation this is because the JDBC driver does not get
> closed on the client side, nor the abstract DBCP connection
> _org.apache.commons.dbcp2.PoolableConnection_ .
> h1. Repro
> (Un)Fortunately I can reproduce the issue using the latest and greatest
> commons-dbcp 2.11.0 and commons-pool 2.12.0 along with OpenJPA 3.2.2.
> I've just created a Java application to reproduce the issue:
> [https://github.com/dionusos/pool_exhausted_repro] . See README.md for
> detailed repro steps.
> h1. Kind of solution?
> To be honest I am not really familiar with DBCP but with this change I
> managed to make my application more robust:
> {code:java}
> diff --git a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> index 440cb756..678550bf 100644
> --- a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> +++ b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> @@ -214,6 +214,10 @@ public class PoolableConnection extends
> DelegatingConnection impleme
> @Override
> protected void handleException(final SQLException e) throws SQLException
> {
> fatalSqlExceptionThrown |= isFatalException(e);
> + if (fatalSqlExceptionThrown && getDelegate() != null) {
> + getDelegate().close();
> + this.close();
> + }
> super.handleException(e);
> }{code}
> What do you think about this approach?
> Is it a completely dead-end or we can start working on it in this direction?
> Do you agree that the reported and reproduced issue is a real one and nut
> just some kind of misconfiguration?
>
> I am lost at this point and I need to move forward so I am asking for
> guidance here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DBCP-590) BasicDataSource#setAbandonedUsageTracking has no effect
[
https://issues.apache.org/jira/browse/DBCP-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz resolved DBCP-590.
--
Fix Version/s: 2.11.1
Resolution: Fixed
I am really sorry it took me so long to figure this out. It looks like indeed
this has been broken since 2.0 and [~ralaoui] is right that the root cause was
BDS not calling use() on the PoolableConnections. I added a test to confirm
that it is working now (as of 4dd9df42ffe0905c6fd74515017a22505ae863cf).
> BasicDataSource#setAbandonedUsageTracking has no effect
> ---
>
> Key: DBCP-590
> URL: https://issues.apache.org/jira/browse/DBCP-590
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Réda Housni Alaoui
>Priority: Major
> Fix For: 2.11.1
>
>
> Passing {{true}} to {{BasicDataSource#setAbandonedUsageTracking(boolean
> usageTracking)}} has no effect because {{UsageTracking#use}} is never called.
> From what I found, {{usageTracking}} can only work if the object pool is of
> type {{ProxiedObjectPool}} . Alas, BasicDataSource enforces
> {{GenericObjectPool}} concrete type preventing us from overriding
> {{BasicDataSource#createObjectPool}} to return a {{ProxiedObjectPool}} .
> Is there something I missed or a workaround?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (DBCP-595) Connection pool can be exhausted when connections are killed on the DB side
[
https://issues.apache.org/jira/browse/DBCP-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816813#comment-17816813
]
Phil Steitz edited comment on DBCP-595 at 2/12/24 11:28 PM:
I just realized there is another scenario that can lead to this, which is due
to a limitation in Commons Pool. If you look at the currently disabled test
case, testLivenessOnTransientFactoryFailure in TestGenericObjectPool, it
illustrates the following scenario:
# Threads enter borrow object when there is capacity to create, but due to
transient factory outage, creates fail.
# The factory starts working again, but threads remain blocked waiting on the
pool
If minIdle > 0 and pool maintenance is enabled (minTimeBetweenEvictionRuns >0),
the pool will create instances when the evictor runs to serve the blocked
threads.
This is being tracked as [POOL-407|
was (Author: psteitz):
I just realized there is another scenario that can lead to this, which is due
to a limitation in Commons Pool. If you look at the currently disabled test
case, testLivenessOnTransientFactoryFailure in TestGenericObjectPool, it
illustrates the following scenario:
# Threads enter borrow object when there is capacity to create, but due to
transient factory outage, creates fail.
# The factory starts working again, but threads remain blocked waiting on the
pool
If minIdle > 0 and pool maintenance is enabled (minTimeBetweenEvictionRuns >0),
the pool will create instances when the evictor runs to serve the blocked
threads.
This is being tracked as
[POOL-407|https://issues.apache.org/jira/projects/POOL/issues/POOL-407?filter=allopenissues
> Connection pool can be exhausted when connections are killed on the DB side
> ---
>
> Key: DBCP-595
> URL: https://issues.apache.org/jira/browse/DBCP-595
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.11.0
>Reporter: Dénes Bodó
>Priority: Critical
> Labels: deadlock, robustness
>
> Apache Oozie 5.2.1 uses OpenJPA 2.4.2 and commons-dbcp 1.4 and commons-pool
> 1.5.4. These are ancient versions, I know.
> h1. Description
> The issue is that when due to some network issues or "maintenance work" on
> the DB side (especially PostgreSQL) which causes the DB connection to be
> closed, it results exhausted Pool on the client side. Many threads are
> waiting at this point:
> {noformat}
> "pool-2-thread-4" #20 prio=5 os_prio=31 tid=0x7faf7903b800 nid=0x8603
> waiting on condition [0x00030f3e7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066aca8e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:1324)
> {noformat}
> According to my observation this is because the JDBC driver does not get
> closed on the client side, nor the abstract DBCP connection
> _org.apache.commons.dbcp2.PoolableConnection_ .
> h1. Repro
> (Un)Fortunately I can reproduce the issue using the latest and greatest
> commons-dbcp 2.11.0 and commons-pool 2.12.0 along with OpenJPA 3.2.2.
> I've just created a Java application to reproduce the issue:
> [https://github.com/dionusos/pool_exhausted_repro] . See README.md for
> detailed repro steps.
> h1. Kind of solution?
> To be honest I am not really familiar with DBCP but with this change I
> managed to make my application more robust:
> {code:java}
> diff --git a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> index 440cb756..678550bf 100644
> --- a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> +++ b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> @@ -214,6 +214,10 @@ public class PoolableConnection extends
> DelegatingConnection impleme
> @Override
> protected void handleException(final SQLException e) throws SQLException
> {
> fatalSqlExceptionThrown |= isFatalException(e);
> + if (fatalSqlExceptionThrown && getDelegate() != null) {
> + getDelegate().close();
> + this.close();
> + }
> super.handleException(e);
> }{code}
> What do you think about this approach?
> Is it a completely dead-end or we can start working on it in this direction?
> Do you agree that the reported and reproduced issue is a real one and nut
> just some kind of misconfiguration?
>
> I am lost at this point and I need to
[jira] [Comment Edited] (DBCP-595) Connection pool can be exhausted when connections are killed on the DB side
[
https://issues.apache.org/jira/browse/DBCP-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816813#comment-17816813
]
Phil Steitz edited comment on DBCP-595 at 2/12/24 11:27 PM:
I just realized there is another scenario that can lead to this, which is due
to a limitation in Commons Pool. If you look at the currently disabled test
case, testLivenessOnTransientFactoryFailure in TestGenericObjectPool, it
illustrates the following scenario:
# Threads enter borrow object when there is capacity to create, but due to
transient factory outage, creates fail.
# The factory starts working again, but threads remain blocked waiting on the
pool
If minIdle > 0 and pool maintenance is enabled (minTimeBetweenEvictionRuns >0),
the pool will create instances when the evictor runs to serve the blocked
threads.
This is being tracked as
[POOL-407|https://issues.apache.org/jira/projects/POOL/issues/POOL-407?filter=allopenissues
was (Author: psteitz):
I just realized there is another scenario that can lead to this, which is due
to a limitation in Commons Pool. If you look at the currently disabled test
case, testLivenessOnTransientFactoryFailure in TestGenericObjectPool, it
illustrates the following scenario:
# Threads enter borrow object when there is capacity to create, but due to
transient factory outage, creates fail.
# The factory starts working again, but threads remain blocked waiting on the
pool
If minIdle > 0 and pool maintenance is enabled (minTimeBetweenEvictionRuns >0),
the pool will create instances when the evictor runs to serve the blocked
threads.
> Connection pool can be exhausted when connections are killed on the DB side
> ---
>
> Key: DBCP-595
> URL: https://issues.apache.org/jira/browse/DBCP-595
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.11.0
>Reporter: Dénes Bodó
>Priority: Critical
> Labels: deadlock, robustness
>
> Apache Oozie 5.2.1 uses OpenJPA 2.4.2 and commons-dbcp 1.4 and commons-pool
> 1.5.4. These are ancient versions, I know.
> h1. Description
> The issue is that when due to some network issues or "maintenance work" on
> the DB side (especially PostgreSQL) which causes the DB connection to be
> closed, it results exhausted Pool on the client side. Many threads are
> waiting at this point:
> {noformat}
> "pool-2-thread-4" #20 prio=5 os_prio=31 tid=0x7faf7903b800 nid=0x8603
> waiting on condition [0x00030f3e7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066aca8e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:1324)
> {noformat}
> According to my observation this is because the JDBC driver does not get
> closed on the client side, nor the abstract DBCP connection
> _org.apache.commons.dbcp2.PoolableConnection_ .
> h1. Repro
> (Un)Fortunately I can reproduce the issue using the latest and greatest
> commons-dbcp 2.11.0 and commons-pool 2.12.0 along with OpenJPA 3.2.2.
> I've just created a Java application to reproduce the issue:
> [https://github.com/dionusos/pool_exhausted_repro] . See README.md for
> detailed repro steps.
> h1. Kind of solution?
> To be honest I am not really familiar with DBCP but with this change I
> managed to make my application more robust:
> {code:java}
> diff --git a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> index 440cb756..678550bf 100644
> --- a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> +++ b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> @@ -214,6 +214,10 @@ public class PoolableConnection extends
> DelegatingConnection impleme
> @Override
> protected void handleException(final SQLException e) throws SQLException
> {
> fatalSqlExceptionThrown |= isFatalException(e);
> + if (fatalSqlExceptionThrown && getDelegate() != null) {
> + getDelegate().close();
> + this.close();
> + }
> super.handleException(e);
> }{code}
> What do you think about this approach?
> Is it a completely dead-end or we can start working on it in this direction?
> Do you agree that the reported and reproduced issue is a real one and nut
> just some kind of misconfiguration?
>
> I am lost at this point and I need to move forward so I am asking for
>
[jira] [Commented] (DBCP-595) Connection pool can be exhausted when connections are killed on the DB side
[
https://issues.apache.org/jira/browse/DBCP-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816813#comment-17816813
]
Phil Steitz commented on DBCP-595:
--
I just realized there is another scenario that can lead to this, which is due
to a limitation in Commons Pool. If you look at the currently disabled test
case, testLivenessOnTransientFactoryFailure in TestGenericObjectPool, it
illustrates the following scenario:
# Threads enter borrow object when there is capacity to create, but due to
transient factory outage, creates fail.
# The factory starts working again, but threads remain blocked waiting on the
pool
If minIdle > 0 and pool maintenance is enabled (minTimeBetweenEvictionRuns >0),
the pool will create instances when the evictor runs to serve the blocked
threads.
> Connection pool can be exhausted when connections are killed on the DB side
> ---
>
> Key: DBCP-595
> URL: https://issues.apache.org/jira/browse/DBCP-595
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.11.0
>Reporter: Dénes Bodó
>Priority: Critical
> Labels: deadlock, robustness
>
> Apache Oozie 5.2.1 uses OpenJPA 2.4.2 and commons-dbcp 1.4 and commons-pool
> 1.5.4. These are ancient versions, I know.
> h1. Description
> The issue is that when due to some network issues or "maintenance work" on
> the DB side (especially PostgreSQL) which causes the DB connection to be
> closed, it results exhausted Pool on the client side. Many threads are
> waiting at this point:
> {noformat}
> "pool-2-thread-4" #20 prio=5 os_prio=31 tid=0x7faf7903b800 nid=0x8603
> waiting on condition [0x00030f3e7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066aca8e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:1324)
> {noformat}
> According to my observation this is because the JDBC driver does not get
> closed on the client side, nor the abstract DBCP connection
> _org.apache.commons.dbcp2.PoolableConnection_ .
> h1. Repro
> (Un)Fortunately I can reproduce the issue using the latest and greatest
> commons-dbcp 2.11.0 and commons-pool 2.12.0 along with OpenJPA 3.2.2.
> I've just created a Java application to reproduce the issue:
> [https://github.com/dionusos/pool_exhausted_repro] . See README.md for
> detailed repro steps.
> h1. Kind of solution?
> To be honest I am not really familiar with DBCP but with this change I
> managed to make my application more robust:
> {code:java}
> diff --git a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> index 440cb756..678550bf 100644
> --- a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> +++ b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> @@ -214,6 +214,10 @@ public class PoolableConnection extends
> DelegatingConnection impleme
> @Override
> protected void handleException(final SQLException e) throws SQLException
> {
> fatalSqlExceptionThrown |= isFatalException(e);
> + if (fatalSqlExceptionThrown && getDelegate() != null) {
> + getDelegate().close();
> + this.close();
> + }
> super.handleException(e);
> }{code}
> What do you think about this approach?
> Is it a completely dead-end or we can start working on it in this direction?
> Do you agree that the reported and reproduced issue is a real one and nut
> just some kind of misconfiguration?
>
> I am lost at this point and I need to move forward so I am asking for
> guidance here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DBCP-595) Connection pool can be exhausted when connections are killed on the DB side
[
https://issues.apache.org/jira/browse/DBCP-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816685#comment-17816685
]
Phil Steitz commented on DBCP-595:
--
Thanks for the info on validation. That info and the partial thread dump above
(can you maybe include a fuller trace?) makes it look likely to me that the
problem is that that the application is not closing connections on some
exception paths. Can you check to make sure that even on exception paths, the
code is calling close on the connection handles returned by DBCP? The fact
that forcing close on fatal exceptions improves things supports the hypothesis
that what is going on is the client code is not closing connections on some
exception paths.
Also, can you confirm that when the starvation happens, the pool is reporting
numActive == maxActive?
With validation turned on, DBCP will reliably close connections when validation
fails. If any kind of exception occurs during the close, DBCP will destroy the
connection handle and reduce its active count. You can see that in the
validateObject method of PoolableConnectionFactory. If you set
fastFailValidation to true, DBCP will fail validation without attempting a
validation query after a fatal exception occurs, again destroying the handle
and adjusting counters.
> Connection pool can be exhausted when connections are killed on the DB side
> ---
>
> Key: DBCP-595
> URL: https://issues.apache.org/jira/browse/DBCP-595
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.11.0
>Reporter: Dénes Bodó
>Priority: Critical
> Labels: deadlock, robustness
>
> Apache Oozie 5.2.1 uses OpenJPA 2.4.2 and commons-dbcp 1.4 and commons-pool
> 1.5.4. These are ancient versions, I know.
> h1. Description
> The issue is that when due to some network issues or "maintenance work" on
> the DB side (especially PostgreSQL) which causes the DB connection to be
> closed, it results exhausted Pool on the client side. Many threads are
> waiting at this point:
> {noformat}
> "pool-2-thread-4" #20 prio=5 os_prio=31 tid=0x7faf7903b800 nid=0x8603
> waiting on condition [0x00030f3e7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066aca8e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:1324)
> {noformat}
> According to my observation this is because the JDBC driver does not get
> closed on the client side, nor the abstract DBCP connection
> _org.apache.commons.dbcp2.PoolableConnection_ .
> h1. Repro
> (Un)Fortunately I can reproduce the issue using the latest and greatest
> commons-dbcp 2.11.0 and commons-pool 2.12.0 along with OpenJPA 3.2.2.
> I've just created a Java application to reproduce the issue:
> [https://github.com/dionusos/pool_exhausted_repro] . See README.md for
> detailed repro steps.
> h1. Kind of solution?
> To be honest I am not really familiar with DBCP but with this change I
> managed to make my application more robust:
> {code:java}
> diff --git a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> index 440cb756..678550bf 100644
> --- a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> +++ b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> @@ -214,6 +214,10 @@ public class PoolableConnection extends
> DelegatingConnection impleme
> @Override
> protected void handleException(final SQLException e) throws SQLException
> {
> fatalSqlExceptionThrown |= isFatalException(e);
> + if (fatalSqlExceptionThrown && getDelegate() != null) {
> + getDelegate().close();
> + this.close();
> + }
> super.handleException(e);
> }{code}
> What do you think about this approach?
> Is it a completely dead-end or we can start working on it in this direction?
> Do you agree that the reported and reproduced issue is a real one and nut
> just some kind of misconfiguration?
>
> I am lost at this point and I need to move forward so I am asking for
> guidance here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DBCP-595) Connection pool can be exhausted when connections are killed on the DB side
[
https://issues.apache.org/jira/browse/DBCP-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816463#comment-17816463
]
Phil Steitz commented on DBCP-595:
--
I am still working through the repro code, but the symptoms don't make sense to
me yet. The stack trace is indicating pool exhaustion, which would mean
clients are holding or abandoning connections. If connections go bad, since
all validation is turned off in the repro code, clients will keep getting bad
connections and their operations will fail; but that does not explain the pool
exhaustion, unless clients are not returning (closing) connections when they
hit the exceptions. Have you tried turning on say testOnReturn? That will
cause bad connections to be closed when they are returned to the pool.
> Connection pool can be exhausted when connections are killed on the DB side
> ---
>
> Key: DBCP-595
> URL: https://issues.apache.org/jira/browse/DBCP-595
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.11.0
>Reporter: Dénes Bodó
>Priority: Critical
> Labels: deadlock, robustness
>
> Apache Oozie 5.2.1 uses OpenJPA 2.4.2 and commons-dbcp 1.4 and commons-pool
> 1.5.4. These are ancient versions, I know.
> h1. Description
> The issue is that when due to some network issues or "maintenance work" on
> the DB side (especially PostgreSQL) which causes the DB connection to be
> closed, it results exhausted Pool on the client side. Many threads are
> waiting at this point:
> {noformat}
> "pool-2-thread-4" #20 prio=5 os_prio=31 tid=0x7faf7903b800 nid=0x8603
> waiting on condition [0x00030f3e7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066aca8e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:1324)
> {noformat}
> According to my observation this is because the JDBC driver does not get
> closed on the client side, nor the abstract DBCP connection
> _org.apache.commons.dbcp2.PoolableConnection_ .
> h1. Repro
> (Un)Fortunately I can reproduce the issue using the latest and greatest
> commons-dbcp 2.11.0 and commons-pool 2.12.0 along with OpenJPA 3.2.2.
> I've just created a Java application to reproduce the issue:
> [https://github.com/dionusos/pool_exhausted_repro] . See README.md for
> detailed repro steps.
> h1. Kind of solution?
> To be honest I am not really familiar with DBCP but with this change I
> managed to make my application more robust:
> {code:java}
> diff --git a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> index 440cb756..678550bf 100644
> --- a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> +++ b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
> @@ -214,6 +214,10 @@ public class PoolableConnection extends
> DelegatingConnection impleme
> @Override
> protected void handleException(final SQLException e) throws SQLException
> {
> fatalSqlExceptionThrown |= isFatalException(e);
> + if (fatalSqlExceptionThrown && getDelegate() != null) {
> + getDelegate().close();
> + this.close();
> + }
> super.handleException(e);
> }{code}
> What do you think about this approach?
> Is it a completely dead-end or we can start working on it in this direction?
> Do you agree that the reported and reproduced issue is a real one and nut
> just some kind of misconfiguration?
>
> I am lost at this point and I need to move forward so I am asking for
> guidance here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-413) [GOP] Race condition while returning objects. maxIdle is ignored
[
https://issues.apache.org/jira/browse/POOL-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772042#comment-17772042
]
Phil Steitz commented on POOL-413:
--
One option to consider here is to use the native capacity management capability
of LinkedBlockingDeque for the final (or other) maxIdle check. We would not
want to raise ISE as the public method does, but instead examine the boolean
returned by LinkFirst/LinkLast That check is made while holding its internal
lock so would be fully threadsafe.
> [GOP] Race condition while returning objects. maxIdle is ignored
>
>
> Key: POOL-413
> URL: https://issues.apache.org/jira/browse/POOL-413
> Project: Commons Pool
> Issue Type: Bug
>Reporter: Adrien Bernard
>Priority: Major
> Attachments:
> 0001-Add-test-to-reproduce-concurrency-issue-when-returni.patch
>
>
> In a GenericObjectPool it is possible to configure a maximum number of idle
> objects to be kept by the pool while they are not in use.
> In unfortunate circumstances, if several threads return an object to the pool
> at the same time, the check on the maximum number of idle objects may be
> dismissed. This results in pool keeping more idle objects than configured.
> I have build a unit test to reproduce the issue. I attach it as a patch made
> on top of release 2.12.0. On my machine it randomly fails with a 10% rate.
> Looking into the source code of the returnObject method of the GOP, it seems
> that there is no synchronisation between the moment the check is made for the
> maxIdle configuration and the moment the object is destroyed :
> {code:java}
> final int maxIdleSave = getMaxIdle();
> if (isClosed() || maxIdleSave > -1 && maxIdleSave <= idleObjects.size()) {
> try {
> destroy(p, DestroyMode.NORMAL);
> } catch (final Exception e) {
> swallowException(e);
> }
> try {
> ensureIdle(1, false);
> } catch (final Exception e) {
> swallowException(e);
> }
> } {code}
> Have you thoughts on this ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-413) [GOP] Race condition while returning objects. maxIdle is ignored
[
https://issues.apache.org/jira/browse/POOL-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772023#comment-17772023
]
Phil Steitz commented on POOL-413:
--
Thanks for reporting this. The analysis looks correct to me. It is possible
for the first check to succeed and then another thread returns an instance
before the first thread adds the instance to the idle pool. Patches welcome
for a fix. One thing to bear in mind: this is a very "hot" method, so we want
to avoid adding locks / sync if possible.
> [GOP] Race condition while returning objects. maxIdle is ignored
>
>
> Key: POOL-413
> URL: https://issues.apache.org/jira/browse/POOL-413
> Project: Commons Pool
> Issue Type: Bug
>Reporter: Adrien Bernard
>Priority: Major
> Attachments:
> 0001-Add-test-to-reproduce-concurrency-issue-when-returni.patch
>
>
> In a GenericObjectPool it is possible to configure a maximum number of idle
> objects to be kept by the pool while they are not in use.
> In unfortunate circumstances, if several threads return an object to the pool
> at the same time, the check on the maximum number of idle objects may be
> dismissed. This results in pool keeping more idle objects than configured.
> I have build a unit test to reproduce the issue. I attach it as a patch made
> on top of release 2.12.0. On my machine it randomly fails with a 10% rate.
> Looking into the source code of the returnObject method of the GOP, it seems
> that there is no synchronisation between the moment the check is made for the
> maxIdle configuration and the moment the object is destroyed :
> {code:java}
> final int maxIdleSave = getMaxIdle();
> if (isClosed() || maxIdleSave > -1 && maxIdleSave <= idleObjects.size()) {
> try {
> destroy(p, DestroyMode.NORMAL);
> } catch (final Exception e) {
> swallowException(e);
> }
> try {
> ensureIdle(1, false);
> } catch (final Exception e) {
> swallowException(e);
> }
> } {code}
> Have you thoughts on this ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed POOL-411.
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (POOL-408) A typo of KeyedPooledObjectFactory on the site and Javadoc
[
https://issues.apache.org/jira/browse/POOL-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed POOL-408.
> A typo of KeyedPooledObjectFactory on the site and Javadoc
> --
>
> Key: POOL-408
> URL: https://issues.apache.org/jira/browse/POOL-408
> Project: Commons Pool
> Issue Type: Bug
>Reporter: Zhenyu Luo
>Priority: Major
> Fix For: 2.12.0
>
> Attachments: screenshot-20220727-173624.png
>
>
> On the homepage, the interface *KeyedPoolableObjectFactory* does not exist.
> The correct interface is {*}KeyedPooledObjectFactory{*}.
> https://commons.apache.org/proper/commons-pool/
> !screenshot-20220727-173624.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (POOL-409) BasicDataSource should support GenericObjectPool->getStatsString()
[
https://issues.apache.org/jira/browse/POOL-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed POOL-409.
> BasicDataSource should support GenericObjectPool->getStatsString()
> --
>
> Key: POOL-409
> URL: https://issues.apache.org/jira/browse/POOL-409
> Project: Commons Pool
> Issue Type: Improvement
>Affects Versions: 2.11.1
>Reporter: Thomas Freller
>Priority: Major
> Labels: improvement
> Fix For: 2.12.0
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Hello,
> I'm developing a Application that is running on a default JRE without an
> Webserver/JMX.
> For optimizing Database connections it would be very useful if i could access
>
> BasicDataSource->GenericObjectPool->{*}getStatsString(){*}
>
> I don't see any reason why this Method is protected and not public in
> GenericObjectPool.
> Then BasicDataSource shoud provide a method getStatsString() or the values
> that represent the statistic data.
>
> If there is any other easy way to access this data within my Java Code I'll
> implement this if you could give me an example how to get this working
> easily. I don't want do configure any jmx stuff.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (POOL-402) Check blockWhenExhausted in hasBorrowWaiters
[ https://issues.apache.org/jira/browse/POOL-402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz closed POOL-402. > Check blockWhenExhausted in hasBorrowWaiters > > > Key: POOL-402 > URL: https://issues.apache.org/jira/browse/POOL-402 > Project: Commons Pool > Issue Type: Improvement >Affects Versions: 2.11.1 >Reporter: Bruno P. Kinoshita >Priority: Major > Fix For: 2.12.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Placeholder for https://github.com/apache/commons-pool/pull/116 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (POOL-394) GenericKeyedObjectPool doesn't have getKeys() implemented
[ https://issues.apache.org/jira/browse/POOL-394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz closed POOL-394. > GenericKeyedObjectPool doesn't have getKeys() implemented > - > > Key: POOL-394 > URL: https://issues.apache.org/jira/browse/POOL-394 > Project: Commons Pool > Issue Type: Bug >Affects Versions: 2.4.2 >Reporter: Vamsi Pavan Kumar Sanka >Priority: Major > Fix For: 2.12.0 > > > getKeys() not really implemented as specified here: > [https://commons.apache.org/proper/commons-pool2/apidocs/org/apache/commons/pool2/impl/GenericKeyedObjectPool.html#getKeys(]) > This API is needed in various scenarios. Current Jira is to expose this API. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (POOL-405) NullPointerException at org.apache.commons.pool2.impl.GenericKeyedObjectPool.invalidateObject(GenericKeyedObjectPool.java:1343)
[
https://issues.apache.org/jira/browse/POOL-405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed POOL-405.
> NullPointerException at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.invalidateObject(GenericKeyedObjectPool.java:1343)
> ---
>
> Key: POOL-405
> URL: https://issues.apache.org/jira/browse/POOL-405
> Project: Commons Pool
> Issue Type: Bug
>Reporter: Gary D. Gregory
>Assignee: Gary D. Gregory
>Priority: Major
> Fix For: 2.12.0
>
>
> You get a {{NullPointerException}} at
> {{org.apache.commons.pool2.impl.GenericKeyedObjectPool.invalidateObject(GenericKeyedObjectPool.java:1343)}}
> when you pass an unknown key.
> The exception should be an {{IllegalStateException}} instead.
> For example:
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.invalidateObject(GenericKeyedObjectPool.java:1343)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.invalidateObject(GenericKeyedObjectPool.java:1320)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (POOL-393) BaseGenericObjectPool.jmxRegister may cost too much time
[
https://issues.apache.org/jira/browse/POOL-393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed POOL-393.
> BaseGenericObjectPool.jmxRegister may cost too much time
>
>
> Key: POOL-393
> URL: https://issues.apache.org/jira/browse/POOL-393
> Project: Commons Pool
> Issue Type: Improvement
>Affects Versions: 2.4.2
>Reporter: Shichao Yuan
>Priority: Major
> Fix For: 2.12.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
>
> When creating many pools, I find that it tasks too much time to register jmx.
> In the code, the ObjectName's postfix always starts with 1, so many
> InstanceAlreadyExistsExceptions may be thrown before registered successfully.
> Maybe a random number is a better choice, or a atomic long.
> {quote}private ObjectName jmxRegister(BaseObjectPoolConfig config,
> String jmxNameBase, String jmxNamePrefix) {
> ObjectName objectName = null;
> MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
> int i = 1;
> boolean registered = false;
> String base = config.getJmxNameBase();
> if (base == null)
> Unknown macro: \{ base = jmxNameBase; }
> while (!registered) {
> try {
> ObjectName objName;
> // Skip the numeric suffix for the first pool in case there is
> // only one so the names are cleaner.
> if (i == 1)
> Unknown macro: \{ objName = new ObjectName(base + jmxNamePrefix); }
> else
> Unknown macro: \{ objName = new ObjectName(base + jmxNamePrefix + i); }
> mbs.registerMBean(this, objName);
> objectName = objName;
> registered = true;
> } catch (MalformedObjectNameException e) {
> if (BaseObjectPoolConfig.DEFAULT_JMX_NAME_PREFIX.equals(
> jmxNamePrefix) && jmxNameBase.equals(base))
> Unknown macro: \{ // Shouldn't happen. Skip registration if it does.
> registered = true; }
> else
> Unknown macro: \{ // Must be an invalid name. Use the defaults instead.
> jmxNamePrefix = BaseObjectPoolConfig.DEFAULT_JMX_NAME_PREFIX; base =
> jmxNameBase; }
> } catch (InstanceAlreadyExistsException e)
> Unknown macro: \{ // Increment the index and try again i++; }
> catch (MBeanRegistrationException e)
> Unknown macro: \{ // Shouldn't happen. Skip registration if it does.
> registered = true; }
> catch (NotCompliantMBeanException e)
> }
> return objectName;
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (POOL-391) GenericKeyedObjectPool is not thread safe when invoke method `borrowObject` and `destroy` simultaneously
[
https://issues.apache.org/jira/browse/POOL-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed POOL-391.
> GenericKeyedObjectPool is not thread safe when invoke method `borrowObject`
> and `destroy` simultaneously
> -
>
> Key: POOL-391
> URL: https://issues.apache.org/jira/browse/POOL-391
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.4.2, 2.5.0, 2.6.0, 2.7.0, 2.8.0, 2.9.0
>Reporter: Codievilky August
>Priority: Blocker
> Fix For: 2.12.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The method `brrowObject` is not thread safe with `destroy` or `clear` method.
> The reason is when use GenericKeyedObjectPool#destroy method，it did not
> ensure the *Atomicity* of destroy object from the ObjectDeque。
> This may cause in the GenericKeyedObjectPool#borrowObject method，may get the
> wrong number of GenericKeyedObjectPool.ObjectDeque#getCreateCount when need
> to create. And the GenericKeyedObjectPool#borrowObject will block until
> timeout.
> Here is the sample test code to recur the bug:
> {code:java}
> // code placeholder
> public class CommonPoolMultiThreadTest {
> public static void main(String[] args) {
> GenericKeyedObjectPoolConfig config = new GenericKeyedObjectPoolConfig();
> config.setMaxTotalPerKey(1);
> config.setMinIdlePerKey(0);
> config.setMaxIdlePerKey(-1);
> config.setMaxTotal(-1);
> config.setMaxWaitMillis(TimeUnit.SECONDS.toMillis(5));
> GenericKeyedObjectPool testPool = new
> GenericKeyedObjectPool<>(
> new KeyedPooledObjectFactory() {
> @Override
> public PooledObject makeObject(Integer key) throws
> Exception {
> System.out.println("start to create");
> return new DefaultPooledObject<>(10);
> } @Override
> public void destroyObject(Integer key, PooledObject p)
> throws Exception {
> System.out.println("start to destroy");
> Thread.sleep(2000);
> } @Override
> public boolean validateObject(Integer key, PooledObject p)
> {
> return true;
> } @Override
> public void activateObject(Integer key, PooledObject p)
> throws Exception {
> // do nothing
> } @Override
> public void passivateObject(Integer key, PooledObject p)
> throws Exception {
> // do nothing
> }
> }, config
> );
> int borrowKey = 10;
> Thread t = new Thread(() -> {
> try {
> while (true) {
> Integer integer = testPool.borrowObject(borrowKey);
> testPool.returnObject(borrowKey, integer);
> Thread.sleep(10);
> }
> } catch (Exception e) {
> e.printStackTrace();
> System.exit(1);
> }
> });
> Thread t2 = new Thread(() -> {
> try {
> while (true) {
> testPool.clear(borrowKey);
> Thread.sleep(10);
> }
> } catch (Exception e) {
> e.printStackTrace();
> System.exit(1);
> }
> });
> t.start();
> t2.start();
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (POOL-242) Set thread name for EvictionTimer
[ https://issues.apache.org/jira/browse/POOL-242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz closed POOL-242. Resolution: Not A Problem > Set thread name for EvictionTimer > - > > Key: POOL-242 > URL: https://issues.apache.org/jira/browse/POOL-242 > Project: Commons Pool > Issue Type: Improvement >Affects Versions: 1.5, 1.6, 2.0, 2.1, 2.2, 2.3 >Reporter: Peter Plosz >Priority: Trivial > > Setting up thread name for EvictionTimer makes life easier when debugging > issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (POOL-299) testOnBorrow overrides testOnCreate=false, add testOnReuse to complement testOnCreate
[
https://issues.apache.org/jira/browse/POOL-299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz updated POOL-299:
-
Fix Version/s: 3.0
> testOnBorrow overrides testOnCreate=false, add testOnReuse to complement
> testOnCreate
> -
>
> Key: POOL-299
> URL: https://issues.apache.org/jira/browse/POOL-299
> Project: Commons Pool
> Issue Type: Improvement
>Affects Versions: 2.4
> Environment: DBCP 2.1.0 with Commons Pool 2.4.0
>Reporter: Justin Cranford
>Priority: Minor
> Labels: performance
> Fix For: 3.0
>
>
> I am using Commons DBCP 2.1 with Commons Pool 2.4. When borrowing a
> connection from DBCP, the underlying pool either reuses a connection or
> creates a new connection. Validation of that connection seems to be triggered
> in borrowObject() of GenericKeyedObjectPool or GenericObjectPool based on two
> settings passed from DBCP 2.1.
> if (p != null && (getTestOnBorrow() || create && getTestOnCreate())) {
> I would like to only validate reused connections, not new connections.
> Validating a new connection is redundant since successful creation of a
> connection implies it is already valid. Commons Pools 2 does not allow me to
> only validate reused connections, though. If I set testOnBorrow=true, then
> Commons Pool 2 validates new or reused connections, and testOnCreate is
> effectively overridden. If I set testOnBorrow=false, then my only option to
> set testOnCreate=true, but that does not match what I need either. No
> combination of these two parameters will work the way I want.
> I would like to offer a possible solution. If Commons Pool 2 were to add a
> new setting called testOnReuse, it would compliment testOnCreate like so:
> >if (p != null && ((getTestOnBorrow()) || (!create &&
> > getTestOnReuse()) || (create && getTestOnCreate( {
> This solution allows testOnBorrow to be backwards compatible - it validates
> both new or reused objects, and it overrides testOnCreate (and testOnReuse).
> However, now I have the option to only validate reused connections, not new
> connections, with this combination of settings:
> - testOnBorrow=false
> - testOnCreate=false
> - testOnReuse=true
> In short, adding testOnReuse would fix the current inability to only validate
> reused objects. It would have the same default value as testOnCreate. If it
> can be added to Commons Pool 2, then I would open an enhancement ticket for
> Commons DBCP 2 to expose it in its programmatic API and config. Any other
> project depending on Commons Pool 2 would benefit as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (POOL-325) testOnReturn() available as asynchronous
[
https://issues.apache.org/jira/browse/POOL-325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz updated POOL-325:
-
Fix Version/s: 3.0
> testOnReturn() available as asynchronous
>
>
> Key: POOL-325
> URL: https://issues.apache.org/jira/browse/POOL-325
> Project: Commons Pool
> Issue Type: New Feature
>Affects Versions: 2.4
>Reporter: Jason Hill
>Priority: Minor
> Labels: performance
> Fix For: 3.0
>
>
> I am using Pool2 as part of DBCP2 framework and I immediately found minor
> performance issues with using testOnBorrow/testOnCreate only because the
> waiting request is dependent on validation step to complete before the
> transaction can complete.
> I opted instead for testWhileIdle and testOnReturn to reduce impact on live
> transactions. I assumed incorrectly that the testOnReturn would be
> asynchronous to prevent the transaction from further waiting on validation
> procedure. I cannot think of a case where testOnReturn is as or more valuable
> than testOnBorrow unless it provided an asynchronous advantage. I am
> interested to hear the intended value?
> Either way, I am proposing that testOnReturn provide asynchronous behavior by
> default or allow a configurable flag to enable async behavior.
> https://git-wip-us.apache.org/repos/asf?p=commons-pool.git;a=blob;f=src/main/java/org/apache/commons/pool2/impl/GenericObjectPool.java;h=816c6cf54aad612805b9d48139a3f87b505c0bd8;hb=HEAD
> CURRENT:
> 548 if (getTestOnReturn()) {
> 549 if (!factory.validateObject(p)) {
> 550 try {
> 551 destroy(p);
> 552 } catch (final Exception e) {
> 553 swallowException(e);
> 554 }
> 555 try {
> 556 ensureIdle(1, false);
> 557 } catch (final Exception e) {
> 558 swallowException(e);
> 559 }
> 560 updateStatsReturn(activeTime);
> 561 return;
> 562 }
> 563 }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (POOL-381) Race condition in the GenericObjectPool in removeAbandoned functionality
[
https://issues.apache.org/jira/browse/POOL-381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed POOL-381.
Resolution: Not A Problem
> Race condition in the GenericObjectPool in removeAbandoned functionality
>
>
> Key: POOL-381
> URL: https://issues.apache.org/jira/browse/POOL-381
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.1
>Reporter: Daniel Dudziak
>Priority: Minor
>
> Hello,
> today I found out a potential issue around GenericObjectPool and removing
> "abandoned" objects functionality.
> During the accident one of our customers faced with database outage.
> After this our product become unreachable. This state was caused by two
> threads.
> 1st "survived" the outage and still possessing the connection it operates on
> the database,
> 2nd one was forced to remove "abandoned" objects.
> This situation cause that 2nd thread was waiting for 1st one until this 1st
> thread end his database operation.
> More context can be found in the stack traces attached below.
> {code:java}
> "1st thread" #434 daemon prio=5 os_prio=0 tid=0x7f9e7746f000 nid=0x9680
> runnable [0x7f9d29c7]"1st thread" #434 daemon prio=5 os_prio=0
> tid=0x7f9e7746f000 nid=0x9680 runnable [0x7f9d29c7]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> - locked <0x7fa23958bc48> (a java.lang.Object)
> at
> oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144)
> at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
> at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
> at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
> at
> oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
> at
> oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:102)
> at
> oracle.jdbc.driver.T4CMAREngineNIO.getNBytes(T4CMAREngineNIO.java:679)
> at
> oracle.jdbc.driver.T4CMAREngineNIO.unmarshalNBytes(T4CMAREngineNIO.java:651)
> at
> oracle.jdbc.driver.DynamicByteArray.unmarshalBuffer(DynamicByteArray.java:334)
> at
> oracle.jdbc.driver.DynamicByteArray.unmarshalCLR(DynamicByteArray.java:226)
> at
> oracle.jdbc.driver.T4CNumberAccessor.unmarshalBytes(T4CNumberAccessor.java:191)
> at
> oracle.jdbc.driver.T4CNumberAccessor.unmarshalOneRow(T4CNumberAccessor.java:173)
> at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:1526)
> at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:1289)
> at oracle.jdbc.driver.T4C8Oall.readRXD(T4C8Oall.java:850)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:543)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
> at
> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)
> at
> oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1023)
> at
> oracle.jdbc.driver.OracleStatement.fetchMoreRows(OracleStatement.java:3353)
> at
> oracle.jdbc.driver.InsensitiveScrollableResultSet.fetchMoreRows(InsensitiveScrollableResultSet.java:736)
> at
> oracle.jdbc.driver.InsensitiveScrollableResultSet.absoluteInternal(InsensitiveScrollableResultSet.java:692)
> at
> oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:406)
> - locked <0x7fa23958e580> (a oracle.jdbc.driver.T4CConnection)
> at
> org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:191)
> at
> org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:191)
> {code}
> {code:java}
> "2nd thread" #376448 daemon prio=5 os_prio=0 tid=0x7f9e581db000
> nid=0x1239 waiting for monitor entry [0x7f9a75ff2000]"2nd thread" #376448
> daemon prio=5 os_prio=0 tid=0x7f9e581db000 nid=0x1239 waiting for monitor
> entry [0x7f9a75ff2000] java.lang.Thread.State: BLOCKED (on object
> monitor) at
> oracle.jdbc.driver.InsensitiveScrollableResultSet.close(InsensitiveScrollableResultSet.java:190)
> - waiting to lock <0x7fa23958e580> at
> org.apache.commons.dbcp2.DelegatingResultSet.close(DelegatingResultSet.java:170)
> at
> org.apache.commons.dbcp2.DelegatingStatement.close(DelegatingStatement.java:150)
> at
>
[jira] [Updated] (POOL-238) Improve test generics
[
https://issues.apache.org/jira/browse/POOL-238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz updated POOL-238:
-
Fix Version/s: 2.13
> Improve test generics
> -
>
> Key: POOL-238
> URL: https://issues.apache.org/jira/browse/POOL-238
> Project: Commons Pool
> Issue Type: Test
>Affects Versions: 1.6, 2.0
>Reporter: Sebb
>Priority: Minor
> Fix For: 2.13
>
>
> From the dev mailing list:
> {noformat}
> > ---
> > commons/proper/pool/trunk/src/test/java/org/apache/commons/pool2/impl/TestGenericKeyedObjectPool.java
> > (original)
> > +++
> > commons/proper/pool/trunk/src/test/java/org/apache/commons/pool2/impl/TestGenericKeyedObjectPool.java
> > Mon Oct 14 20:42:03 2013
> > @@ -60,10 +60,10 @@ public class TestGenericKeyedObjectPool
> > protected KeyedObjectPool makeEmptyPool(int
> > mincapacity) {
> >
> > KeyedPooledObjectFactory factory =
> > -new KeyedPooledObjectFactory() {
> > +new BaseKeyedPooledObjectFactory() {
> One day, if someone wants to do a public service and seize a great
> opportunity to jump into [pool], s/he will fix the cheezy Object> setup in these tests and actually generify the test
> factories here. :)
> Phil
> {noformat}
> Created issue so it's not forgotten
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (POOL-353) Return false if current connection count is less then MinIdle in DefaultEvictionPolicy
[ https://issues.apache.org/jira/browse/POOL-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz updated POOL-353: - Fix Version/s: (was: 2.12.0) > Return false if current connection count is less then MinIdle in > DefaultEvictionPolicy > -- > > Key: POOL-353 > URL: https://issues.apache.org/jira/browse/POOL-353 > Project: Commons Pool > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: jefferyyuan >Assignee: Mark Struberg >Priority: Minor > Fix For: 3.0 > > > At > [https://github.com/apache/commons-pool/blob/master/src/main/java/org/apache/commons/pool2/impl/BaseGenericObjectPool.java#L1140] > It first evicts idle connections, then re-creates idle instances to > ensureMinIdle. > DefaultEvictionPolicy will close idle connection even when there is <= > MinIdle connections. > https://github.com/apache/commons-pool/blob/master/src/main/java/org/apache/commons/pool2/impl/DefaultEvictionPolicy.java > > In case minIdle is set to large, it would causes problems like frequent full > GC. > * the connection related objects are promoted and stored in old gen, then > evict will close them and make it gc-able, and ensureMinIdle will create new > connections which will be eventually promoted to old gen and gc-able again. > * Old gen will grow quickly and need full gc. > We can solve the problem if we always keep minIdle connections in the > DefaultEvictionPolicy. > Or at least add doc to DefaultEvictionPolicy and may provide another > EvictionPolicy which returns false if current connection count is less then > MinIdle and refer it in DefaultEvictionPolicy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (POOL-400) Refactor TestGenericObjectPoolFactoryCreateFailure to improve test design
[
https://issues.apache.org/jira/browse/POOL-400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed POOL-400.
Resolution: Fixed
I do not see the need to bring in a new dependency for this case.
> Refactor TestGenericObjectPoolFactoryCreateFailure to improve test design
> -
>
> Key: POOL-400
> URL: https://issues.apache.org/jira/browse/POOL-400
> Project: Commons Pool
> Issue Type: Improvement
>Reporter: Xiao Wang
>Priority: Minor
>
> h3. Description
> I noticed that there is a test class
> [SingleObjectFactory|https://github.com/apache/commons-pool/blob/dbb4ca0bc969464f31435867a0800fc766f31068/src/test/java/org/apache/commons/pool2/impl/TestGenericObjectPoolFactoryCreateFailure.java#L38]
> extends production class
> [BasePooledObjectFactory|https://github.com/apache/commons-pool/blob/dbb4ca0bc969464f31435867a0800fc766f31068/src/main/java/org/apache/commons/pool2/BasePooledObjectFactory.java#L33]
> to assist testing method
> [GenericObjectPool.borrowObject()|https://github.com/apache/commons-pool/blob/dbb4ca0bc969464f31435867a0800fc766f31068/src/main/java/org/apache/commons/pool2/impl/GenericObjectPool.java#L222].
> This might not be the best priactice in unit testing and can be improved by
> leveraging mocking frameworks.
> h3. Current Implementation
> * {{SingleObjectFactory}} extends {{BasePooledObjectFactory}} and creates a
> new variable to keep tracking of the method invocation status for
> {{created()}}.
> * In test case, the new variable will be used to check the execution status
> and wait {{created()}} to be excuted.
> h3. Proposed Implementation
> * Replace {{SingleObjectFactory}} with a mocking object created by Mockito.
> * Extract the AtomicBoolean attribute and use the extracted attribute in
> test case to check method invocation status.
> * Use method stub to control the behavior of the mocking object.
> h3. Motivation
> * Decouple test class {{SingleObjectFactory}} from production interface
> {{BasePooledObjectFactory}}.
> * Make test logic more clear by using method stub instead of method
> overriding.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (POOL-282) CLONE - Support AbandonedConfig in SharedPoolDataSource
[ https://issues.apache.org/jira/browse/POOL-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz updated POOL-282: - Fix Version/s: 2.13 > CLONE - Support AbandonedConfig in SharedPoolDataSource > --- > > Key: POOL-282 > URL: https://issues.apache.org/jira/browse/POOL-282 > Project: Commons Pool > Issue Type: Improvement >Affects Versions: 2.0, 2.1, 2.2, 2.3 > Environment: Linux and Java 1.6 >Reporter: Michael Kerr >Priority: Minor > Labels: features > Fix For: 2.13 > > > BasicDataSource has support for AbandonedConfig > SharedPoolDataSource does not currently support reclaiming abandoned > connections. > It would be helpful if the user can extend the underlying connection pool to > add features like abandoned connection management. This is possible with > BasicDataSource. > Currently the only solution is to extend classes or replace classes in the > Apache package namespace. > Thanks for looking, > Michael -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (POOL-290) TestSoftRefOutOfMemory goes into infinite loop
[
https://issues.apache.org/jira/browse/POOL-290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz updated POOL-290:
-
Fix Version/s: 2.13
> TestSoftRefOutOfMemory goes into infinite loop
> --
>
> Key: POOL-290
> URL: https://issues.apache.org/jira/browse/POOL-290
> Project: Commons Pool
> Issue Type: Bug
>Reporter: Serge Angelov
>Priority: Major
> Fix For: 2.13
>
>
> When running TestSoftOutOfMemory tests, application goes into infinite loop
> in case if OutOfMemoryError happens. It happens for each function that tests
> OutOfMemory case.
> For example in this code
> {code}
> while (pool.getNumIdle() > 0) {
> try {
> long freeMemory = runtime.freeMemory();
> if (freeMemory > Integer.MAX_VALUE) {
> freeMemory = Integer.MAX_VALUE;
> }
> garbage.add(new byte[Math.min(1024 * 1024,
> (int)freeMemory/2)]);
> } catch (OutOfMemoryError oome) {
> System.gc();
> }
> System.gc();
> }
> {code}
> Correct me if I'm wrong, but wouldn't it be more efficient to destroy pool
> immediatelly if error happens and wait till we free that piece of memory (for
> example wait till free memory increased by the size of current pool).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (POOL-351) Add JMX Notifications for NumActive and NumIdle
[ https://issues.apache.org/jira/browse/POOL-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz updated POOL-351: - Fix Version/s: 3.0 > Add JMX Notifications for NumActive and NumIdle > --- > > Key: POOL-351 > URL: https://issues.apache.org/jira/browse/POOL-351 > Project: Commons Pool > Issue Type: New Feature >Reporter: Paul Bakker >Priority: Major > Fix For: 3.0 > > > Through JMX you can currently get the NumActive and NumIdle. These two values > are constantly changing though, which means if you want to monitor them and > get accurate stats, you need to read these values very often. > In order to minimize the polling frequency, I'd rather have the ability to > get notified if the numbers change -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (POOL-305) Add maxAge to pool config
[
https://issues.apache.org/jira/browse/POOL-305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz updated POOL-305:
-
Fix Version/s: 3.0
> Add maxAge to pool config
> -
>
> Key: POOL-305
> URL: https://issues.apache.org/jira/browse/POOL-305
> Project: Commons Pool
> Issue Type: Improvement
>Affects Versions: 2.4.2
>Reporter: Michael Osipov
>Priority: Major
> Fix For: 3.0
>
>
> In our environment, there are upper caps to session lifetime regardless how
> often the session has been busy/idle. Given that an object in a pool can
> remain for an indefinite time as long as it is used often enough, it may also
> already be invalided by the server.
> To avoid such issues, I'd like to have a {{maxAge}} property for an object,
> where eviction kicks in by default. This is the very same as in the [Tomcat
> JDBC Connection Pool|https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html].
> I am quite certain that this is possible with a custom eviction policy but I
> do think that should be available by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (POOL-321) ObjectPool.addObject limiting us to add an object with parameter
[ https://issues.apache.org/jira/browse/POOL-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz updated POOL-321: - Fix Version/s: 3.0 (was: 2.12.0) > ObjectPool.addObject limiting us to add an object with parameter > > > Key: POOL-321 > URL: https://issues.apache.org/jira/browse/POOL-321 > Project: Commons Pool > Issue Type: Improvement >Affects Versions: 2.4 >Reporter: Ugur Bayram >Priority: Major > Fix For: 3.0 > > > In current implementation we are not able to add an object with parameter to > the pool. addObject does not take any parameter > Consider of a user pool where the content of each user might differ. In such > cases, current implementation have limitation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (POOL-267) Use of AbandonListener instead of PrintWriter when a pool object is detected in an abandon state
[ https://issues.apache.org/jira/browse/POOL-267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz updated POOL-267: - Fix Version/s: 2.13 (was: 2.12.0) > Use of AbandonListener instead of PrintWriter when a pool object is detected > in an abandon state > > > Key: POOL-267 > URL: https://issues.apache.org/jira/browse/POOL-267 > Project: Commons Pool > Issue Type: Improvement >Affects Versions: 2.2 >Reporter: Anthony Communier >Priority: Major > Fix For: 2.13 > > Attachments: AbandonedListenerPatch.diff > > > I would be great to have a more flexible mecanism for abandon detection. > Having just a PrintWriter is too restrictive. With an AbandonListener it > would be possible to make more things like using log API or enabling > supervision of application. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (POOL-272) GenericKeyedObjectPool should have a per-key version of numTestsPerEvictionRun
[ https://issues.apache.org/jira/browse/POOL-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz updated POOL-272: - Fix Version/s: 2.13 (was: 2.12.0) > GenericKeyedObjectPool should have a per-key version of numTestsPerEvictionRun > -- > > Key: POOL-272 > URL: https://issues.apache.org/jira/browse/POOL-272 > Project: Commons Pool > Issue Type: Improvement >Affects Versions: 2.2 >Reporter: Michael Berman >Priority: Major > Fix For: 2.13 > > > It would be cool if there were a per-key version of numTestsPerEvictionRun. I > have a use case where I'm primarily configuring my pool size (min/maxidle and > maxtotal) per key, but don't want to cap the number of keys or total number > of objects in the pool. In this case, it seems like the number of tests i > would want per eviction run is fixed with respect to my per-key pool size, > but that I would want it scale with the number of keys. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (POOL-406) add support for Micrometer
[ https://issues.apache.org/jira/browse/POOL-406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz updated POOL-406: - Fix Version/s: 3.0 > add support for Micrometer > --- > > Key: POOL-406 > URL: https://issues.apache.org/jira/browse/POOL-406 > Project: Commons Pool > Issue Type: New Feature >Reporter: JoshuaXu >Priority: Major > Fix For: 3.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > monitor the pool status will be useful in the prodution environment -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (POOL-372) CallStackUtils mishandles security manager check part 2
[ https://issues.apache.org/jira/browse/POOL-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz updated POOL-372: - Fix Version/s: 3.0 > CallStackUtils mishandles security manager check part 2 > --- > > Key: POOL-372 > URL: https://issues.apache.org/jira/browse/POOL-372 > Project: Commons Pool > Issue Type: Bug >Reporter: Volker Kleinschmidt >Priority: Major > Fix For: 3.0 > > > This ticket is for (b). > CallStackUtils determines at initialization time whether it is allowed to > create a security manager, then sticks that info into a static variable and > never checks it again, relying on this check to later try to create a > SecurityManager for a SecurityManagerCallStack. This is doubly wrong: > a) If the code is running in a privileged context at init time, it determines > that it can create a security manager, and then later naively assumes that > henceforth all code is privileged and also can create a security manager. Of > course this is not true, otherwise one would not need a security manager in > the first place! This info can never be kept in a static variable, it's > extremely context-dependent. So this leads to AccessControlException from > invoking newCallStack() if abandoned object logging is enabled. > b) The permission to create a security manager must never be granted to any > code, unless that code has AllPermission in the first place, i.e. is already > fully privileged. This is because this permission allows circumventing the > security manager completely (simply create one that lets all checks pass). > Therefore even just checking whether you're allowed to create a secmgr is > naive - if a secmgr is installed at all you should assume that you're NOT > privileged enough to do this, simply because for sure some code that calls > your code will not be privileged enough. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (POOL-401) GKOP invalidateObject should make capacity available to all keyed pools
[ https://issues.apache.org/jira/browse/POOL-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz updated POOL-401: - Fix Version/s: 2.13 > GKOP invalidateObject should make capacity available to all keyed pools > --- > > Key: POOL-401 > URL: https://issues.apache.org/jira/browse/POOL-401 > Project: Commons Pool > Issue Type: Bug >Reporter: Phil Steitz >Priority: Major > Fix For: 2.13 > > > GKOP invalidateObject currently only creates and adds objects if there are > borrowers waiting for the keyed pool that the instance being invalidated came > from. If there are no take waiters on that pool, or there is a more heavily > loaded pool, the (total) capacity should be reused in a different pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (POOL-407) Threads get stuck when idleObjects list is empty.
[
https://issues.apache.org/jira/browse/POOL-407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz updated POOL-407:
-
Fix Version/s: 3.0
> Threads get stuck when idleObjects list is empty.
> -
>
> Key: POOL-407
> URL: https://issues.apache.org/jira/browse/POOL-407
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Sarthak Shukla
>Priority: Major
> Fix For: 3.0
>
>
> While borrowing object from pool, threads are getting stuck. I initialised
> the pool size as 1. And had 3 threads created. First thread enters
> borrowObject method, since there are no idle objects to poll from, it will
> create one object and move forward.
> {code:java}
> p = (PooledObject)this.idleObjects.pollFirst();
> if (p == null) {
> p = this.create();
> if (p != null) {
> create = true;
> }
> } {code}
> The other two threads will also follow same path and check for idle
> objects(there are none), will try to create one object but the pool size is
> set to 1. Thus, the two threads will move forward and enter
> *idleObjects.takeFirst()* function. Value of blockWhenExhausted is true and
> borrowMaxWaitMillis is -1 as we don't want timeout.
> {code:java}
> if (blockWhenExhausted) {
>if (p == null) {
> if (borrowMaxWaitMillis < 0L) {
>p = (PooledObject)this.idleObjects.takeFirst();
> } else {
>p = (PooledObject)this.idleObjects.pollFirst(borrowMaxWaitMillis,
> TimeUnit.MILLISECONDS);
> }
>}
>if (p == null) {
> throw new NoSuchElementException("Timeout waiting for idle object");
>}
> }{code}
> Now, the main thread does *this.factory.activateObject(p);* and object gets
> activated. Now, when the validation is checked *validate =
> this.factory.validateObject(p);* it comes out to be false as provider might
> have been disconnected.
> So, the object is destroyed by calling *this.destroy(p);*
> {code:java}
> private void destroy(PooledObject toDestroy) throws Exception {
> toDestroy.invalidate();
> this.idleObjects.remove(toDestroy);
> this.allObjects.remove(new
> BaseGenericObjectPool.IdentityWrapper(toDestroy.getObject()));
> try {
> this.factory.destroyObject(toDestroy);
> } finally {
> this.destroyedCount.incrementAndGet();
> this.createCount.decrementAndGet();
> }
> }{code}
> The object which was created is now destroyed and removed from idleObject and
> allObjects list. Now, the other two threads are still waiting to take object
> from idle objects list but there are no object present. Hence, the two
> threads are in wait state for infinite period and the application waits
> forever until we kill the process.
> {code:java}
> public E takeFirst() throws InterruptedException {
>this.lock.lock();
>Object var2;
>try {
> Object x;
> while((x = this.unlinkFirst()) == null) {
> this.notEmpty.await();
> }
> var2 = x;
> } finally {
> this.lock.unlock();
> }
> return var2;
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (POOL-410) EHN Max Concurrent Connections in Stats
[ https://issues.apache.org/jira/browse/POOL-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Steitz updated POOL-410: - Fix Version/s: 2.13 > EHN Max Concurrent Connections in Stats > --- > > Key: POOL-410 > URL: https://issues.apache.org/jira/browse/POOL-410 > Project: Commons Pool > Issue Type: Improvement >Reporter: Thomas Freller >Priority: Major > Fix For: 2.13 > > > For tuning the Database the max concurrent used connections wold be relay > great. > > So I could monitor the max concurrent connections of my pool and could see if > I have to size up/down the max database connections (that can me memory > intensive) or if I could downsize the Pool and also the Database i could save > Memory and costs. At the Moment the only Indicator is > maxBorrowWaitDuration=PT3.016S, > > But this can only tell me hat at the Moment the pool reached its max (because > I set maxTotal to 1) my normal Case is to have up to 20 Pools of different > Applications that connect to one instance of MariaDB and the output of > maxBorrowWaitDuration is 0. So I only know I would be able to downsize the > pool but I have no hint how to size the pool best. > This Statistical Data should be also printed in the toString() Method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (POOL-350) Add option for not executing "hasBorrowWaiters()" while returning objects
[
https://issues.apache.org/jira/browse/POOL-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz updated POOL-350:
-
Fix Version/s: 2.13
> Add option for not executing "hasBorrowWaiters()" while returning objects
> -
>
> Key: POOL-350
> URL: https://issues.apache.org/jira/browse/POOL-350
> Project: Commons Pool
> Issue Type: New Feature
>Affects Versions: 2.6.0
> Environment: h5. uname -a:
> Linux VMS26239 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015
> x86_64 x86_64 x86_64 GNU/Linux
>
> *Java version:*
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>
>
>Reporter: zhu chen
>Priority: Critical
> Labels: easyfix
> Fix For: 2.13
>
>
> h2. Phenomena:
> I'm recently leveraging commons-pool as my Redis connection pool in my
> project, however, the pain is that when my system is dealing with over
> thousands of Redises, CPU load become such high. By checking JVM through
> JFR(FlightRecorder), it turned out the hot method was
> "{color:#FF}hasBorrowWaiters(){color}", which is invoked by
> "{color:#FF}returnObject(){color}" each time.
> That means the system will go through over *thousands*(the number will grow
> as well as my system) of keys after *each* object's *return*, what's worse,
> the program is running concurrently, which, obviously cause a huge CPU load.
>
> h2. Expect:
> I was wondering if we could add a config for optionally run this
> "hasBorrowWaiters()" each time when we return an object.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (POOL-269) Use generic exceptions instead of java.lang.Exception
[
https://issues.apache.org/jira/browse/POOL-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz updated POOL-269:
-
Fix Version/s: 3.0
(was: 2.12.0)
> Use generic exceptions instead of java.lang.Exception
> -
>
> Key: POOL-269
> URL: https://issues.apache.org/jira/browse/POOL-269
> Project: Commons Pool
> Issue Type: Improvement
>Affects Versions: 2.2
>Reporter: Michael Osipov
>Assignee: Gary D. Gregory
>Priority: Major
> Fix For: 3.0
>
>
> Too many methods say {{throws Exception}} in their signature. This is neither
> helpful nor good API design. You never know what the exception is and where
> it came from.
> An exception translation pattern has to be applied to make code usable, e.g.,
> like the Spring project does or Maven with {{BuildException}}.
> Unfortunately, this ugly practive has prevailed in several Apache projects
> like Lucene.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz resolved POOL-411.
--
Resolution: Fixed
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749423#comment-17749423
]
Phil Steitz commented on POOL-411:
--
Fully fixed in 980aae56bf732178a9e02a3cb686f0104bffb6d3.
The problem leading to this and other GKOP failures when pools are rapidly
created and destroyed was that the pool retrieved from the poolMap at the
beginning of register and deregister could be replaced between the time the
read lock is released and the write lock is obtained, resulting in the counter
update after write lock acquisition being made to the wrong pool.
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-410) EHN Max Concurrent Connections in Stats
[ https://issues.apache.org/jira/browse/POOL-410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743558#comment-17743558 ] Phil Steitz commented on POOL-410: -- Moving a comment from PR to here. I think I now understand at least the stats / reporting ask here. We currently provide stats caching and reporting for timing metrics. I think it is reasonable to ask for that for active and idle counts. I would be happy to review a PR to add activeCounts and idleCounts to go along with activeTimes and the other latency StatsStore instances now in BaseGenericObjectPool. I would approach that by creating activeCounts and idleCounts as StatsStore instances in BGOP and updating them in updateStatsOnReturn and updateStatsOnBorrow, using current active and idle counts from the pool. Then include these stats in reports. And tests. It is maybe a slippery slope to add more and more stats caching and It might actually be better to move in 3.0 to stats logging or listeners, but I think idle/active counts are important enough to be added to what is currently being cached. Another thing that is missing is control over the cache window / resetting the caches. For the OPs use case, some way to set the cache size or time to live would be needed. As of 2.11 it is hard-coded to the 100 most recent measurements. Adding cache size confguration and a method to clear the cache would be helpful here. > EHN Max Concurrent Connections in Stats > --- > > Key: POOL-410 > URL: https://issues.apache.org/jira/browse/POOL-410 > Project: Commons Pool > Issue Type: Improvement >Reporter: Thomas Freller >Priority: Major > > For tuning the Database the max concurrent used connections wold be relay > great. > > So I could monitor the max concurrent connections of my pool and could see if > I have to size up/down the max database connections (that can me memory > intensive) or if I could downsize the Pool and also the Database i could save > Memory and costs. At the Moment the only Indicator is > maxBorrowWaitDuration=PT3.016S, > > But this can only tell me hat at the Moment the pool reached its max (because > I set maxTotal to 1) my normal Case is to have up to 20 Pools of different > Applications that connect to one instance of MariaDB and the output of > maxBorrowWaitDuration is 0. So I only know I would be able to downsize the > pool but I have no hint how to size the pool best. > This Statistical Data should be also printed in the toString() Method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (POOL-410) EHN Max Concurrent Connections in Stats
[ https://issues.apache.org/jira/browse/POOL-410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743460#comment-17743460 ] Phil Steitz commented on POOL-410: -- This looks more like a question for the user list. For monitoring, you already have numActive (number checked out) and numIdle (number idle in the pool) exposed. And you can set maxIdle. I don't understand fully what you are trying to do. The maxTotal property does what I suspect your real need is - control the total number of provider resources that are dedicated to / under management of the pool. That will bound the maximum number of concurrent instances checked out to clients. One more note: if what you are pooling are database connections, Commons DBCP makes that easier to setup and manage and exposes some more jdbc-specific properties. > EHN Max Concurrent Connections in Stats > --- > > Key: POOL-410 > URL: https://issues.apache.org/jira/browse/POOL-410 > Project: Commons Pool > Issue Type: Improvement >Reporter: Thomas Freller >Priority: Major > > For tuning the Database the max concurrent used connections wold be relay > great. > > So I could monitor the max concurrent connections of my pool and could see if > I have to size up/down the max database connections (that can me memory > intensive) or if I could downsize the Pool and also the Database i could save > Memory and costs. At the Moment the only Indicator is > maxBorrowWaitDuration=PT3.016S, > > But this can only tell me hat at the Moment the pool reached its max (because > I set maxTotal to 1) my normal Case is to have up to 20 Pools of different > Applications that connect to one instance of MariaDB and the output of > maxBorrowWaitDuration is 0. So I only know I would be able to downsize the > pool but I have no hint how to size the pool best. > This Statistical Data should be also printed in the toString() Method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (POOL-407) Threads get stuck when idleObjects list is empty.
[
https://issues.apache.org/jira/browse/POOL-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742216#comment-17742216
]
Phil Steitz commented on POOL-407:
--
Looking more carefully at this, the sequence in the description can happen and
it is a liveness limitation of the pool. There are other scenarios that can
lead to this. Neither GKOP nor GOP currently have a way to recover from
factory "outages." My suggestion to use ensureIdle above is not a good idea
because it could lead to looping in borrowObject.
The tests added above are not really testing anything. You can see in the
console NPE stack traces that don't cause the Junit tests to fail because the
runner is not picking them up. A more complex setup would be needed to catch
them. That setup would cause the ones where the factory returns null to fail.
By design, pool create methods throw NPE when factories return null. This is
documented in the javadoc for create, but missing from that of the pool
methods.
> Threads get stuck when idleObjects list is empty.
> -
>
> Key: POOL-407
> URL: https://issues.apache.org/jira/browse/POOL-407
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Sarthak Shukla
>Priority: Major
>
> While borrowing object from pool, threads are getting stuck. I initialised
> the pool size as 1. And had 3 threads created. First thread enters
> borrowObject method, since there are no idle objects to poll from, it will
> create one object and move forward.
> {code:java}
> p = (PooledObject)this.idleObjects.pollFirst();
> if (p == null) {
> p = this.create();
> if (p != null) {
> create = true;
> }
> } {code}
> The other two threads will also follow same path and check for idle
> objects(there are none), will try to create one object but the pool size is
> set to 1. Thus, the two threads will move forward and enter
> *idleObjects.takeFirst()* function. Value of blockWhenExhausted is true and
> borrowMaxWaitMillis is -1 as we don't want timeout.
> {code:java}
> if (blockWhenExhausted) {
>if (p == null) {
> if (borrowMaxWaitMillis < 0L) {
>p = (PooledObject)this.idleObjects.takeFirst();
> } else {
>p = (PooledObject)this.idleObjects.pollFirst(borrowMaxWaitMillis,
> TimeUnit.MILLISECONDS);
> }
>}
>if (p == null) {
> throw new NoSuchElementException("Timeout waiting for idle object");
>}
> }{code}
> Now, the main thread does *this.factory.activateObject(p);* and object gets
> activated. Now, when the validation is checked *validate =
> this.factory.validateObject(p);* it comes out to be false as provider might
> have been disconnected.
> So, the object is destroyed by calling *this.destroy(p);*
> {code:java}
> private void destroy(PooledObject toDestroy) throws Exception {
> toDestroy.invalidate();
> this.idleObjects.remove(toDestroy);
> this.allObjects.remove(new
> BaseGenericObjectPool.IdentityWrapper(toDestroy.getObject()));
> try {
> this.factory.destroyObject(toDestroy);
> } finally {
> this.destroyedCount.incrementAndGet();
> this.createCount.decrementAndGet();
> }
> }{code}
> The object which was created is now destroyed and removed from idleObject and
> allObjects list. Now, the other two threads are still waiting to take object
> from idle objects list but there are no object present. Hence, the two
> threads are in wait state for infinite period and the application waits
> forever until we kill the process.
> {code:java}
> public E takeFirst() throws InterruptedException {
>this.lock.lock();
>Object var2;
>try {
> Object x;
> while((x = this.unlinkFirst()) == null) {
> this.notEmpty.await();
> }
> var2 = x;
> } finally {
> this.lock.unlock();
> }
> return var2;
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741435#comment-17741435
]
Phil Steitz commented on POOL-411:
--
[~sebb] I added sleeps temporarily between the locks in register/deregister and
a couple of timing-related tests failed because the latency controls in the
tests did not work correctly, but nothing else failed. I am not sure how to
elegantly set this up so it can be done periodically, but I agree it is a good
idea to do maybe in the runup to releases (like running soak and performance
tests with commons-performance).
On the double-checking, there are two cases to consider: first, register.
There is basically double-checking there or more precisely assurance provided
by computeIfAbsent. That is really where the bug fix is. It makes sure that
if there was no pool under the key when the read lock was acquired and there is
one after the write lock is acquired, no new pool is created. Second is
deregister. If poolMap.get(k) returns null after getting the write lock that
means that the original pool has been removed and we are in a deregister
applied to the wrong pool situation. The right thing to do there is throw ISE,
but it should not be able to happen (which is why there was no null check
before this bug was reported). We could add a test there, but I am concerned
that it would be a check that should never be able to fail and there is small,
but not zero performance cost to performing it for pretty much every pool
action (register/deregister are very hot methods). The same applies to testing
the counters on every inspection.
Perhaps best is to temporarily insert these checks along with the sleeps as
part of test runs. Any suggestions about how to do that in a simple,
maintainable way?
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741331#comment-17741331
]
Phil Steitz edited comment on POOL-411 at 7/8/23 9:16 PM:
--
[~sebb] I am not sure I understand your last comment. With the test parms (now
committed) above, the test fails regularly before Gary's last commit, so I am
not sure we need to add the sleeps to verify. I agree with what I think you
are implying - that the problem in prior code was likely between the read lock
release and write lock acquire, resulting in the possibility of two registering
threads ending up creating two different pools under the same key, one of which
ends up orphaned. More importantly, the deregister from one thread ends up
decrementing numInterested on a different deque from the one it created but
orphaned in its register activation, resulting in its premature demise. That
would lead to exactly what we saw in prior code. Gary's change effectively
double checks that the pool exists before adding to the poolMap. There is no
need to null check the deque because the computeIfAbsent creates it if it is
not there. I think we are good to resolve this unless I am missing something.
was (Author: psteitz):
[~sebb] I am not sure I understand your last comment. With the test parms (now
committed) above, the test fails regularly before Gary's last commit, so I am
not sure we need to add the sleeps to verify. I agree with what I think you
are implying - that the problem in prior code was likely between the read lock
release and write lock acquire, resulting in the possibility of two registering
threads ending up creating two different pools under the same key, one of which
ends up orphaned. Gary's change effectively double checks that the pool exists
before adding to the poolMap. I haven't looked at the ConcurrentHashMap
sources to verify, but I suspect that you can't have two threads concurrently
executing the lambda, so this should fully close the hole. There is no need
to null check the deque because the computeIfAbsent creates it if it is not
there. I think we are good to resolve this unless I am missing something.
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741331#comment-17741331
]
Phil Steitz commented on POOL-411:
--
[~sebb] I am not sure I understand your last comment. With the test parms (now
committed) above, the test fails regularly before Gary's last commit, so I am
not sure we need to add the sleeps to verify. I agree with what I think you
are implying - that the problem in prior code was likely between the read lock
release and write lock acquire, resulting in the possibility of two registering
threads ending up creating two different pools under the same key, one of which
ends up orphaned. Gary's change effectively double checks that the pool exists
before adding to the poolMap. I haven't looked at the ConcurrentHashMap
sources to verify, but I suspect that you can't have two threads concurrently
executing the lambda, so this should fully close the hole. There is no need
to null check the deque because the computeIfAbsent creates it if it is not
there. I think we are good to resolve this unless I am missing something.
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741220#comment-17741220
]
Phil Steitz commented on POOL-411:
--
[~ggregory] - looks great. Great idea to use computeIfAbsent and to make the
read on numInterested atomic. I wish I understood better the sequence that
gets the counter off, but this looks like it should help. One small comment:
the allocate variable added in register can just be a normal boolean, since it
is only visible to the executing thread. I would say go ahead and commit this
change and we can keep testing.
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741156#comment-17741156
]
Phil Steitz commented on POOL-411:
--
If I make the following changes to testConcurrentBorrowAnd clear, the test
fails about one in three times against the code in master now (as of
108735e2e262bad80f1f18abe9feae93b5db0624)
# Reduce threadCount and taskCount to 2
# Increase borrow and clear cycles to 5000
# Eliminate Thread.yields
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741106#comment-17741106
]
Phil Steitz edited comment on POOL-411 at 7/7/23 3:50 PM:
--
[~sebb] - thanks for looking at this. You are right - the scenario I posted
(and deleted) is impossible. The jdk docs are a little vague, but I have
confirmed you are right - the thread trying to get the write lock will have to
block until any readers release their read locks. The null guard that was
added in deregister prevents the exception reported above from happening, but
the test case that Gary added now fails sporadically (very rarely) with this in
the stack trace:
{{java.lang.NullPointerException: Cannot invoke
"org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getIdleObjects()"
because the return value of "java.util.Map.get(Object)" is null}}
{{ at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.addIdleObject(GenericKeyedObjectPool.java:308)}}
{{ at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:333)}}
{{ at org.apache.com}}
The problem with above is that addObject calls register on the key before
invoking addIdleObject so any deregister that happens in between should not be
able to delete the pool.
The intent is for register to signal that a thread is "interested" in a key so
others should refrain from deleting it. It will also create an empty pool if
there is no pool under the given key. It returns the pool under the registered
key. Calling deregister says the thread is no longer interested and also
checks to see if the associated keyed pool is empty, has no objects under
management and has no other threads interested in it. In that case, it deletes
it. This setup ensures that a) when you call register, you can be sure that
you will get a non-null pool and that pool will not be deleted until after you
deregister it b) empty pools with no objects under management and no interested
threads get cleaned up. Thread interest is tracked in the numInterested counter
attached to the pool itself, so what must be going on is that counter must be
getting corrupted somehow. One way that could happen is if deregister is
called twice for one registration.}}{}}}
was (Author: psteitz):
[~sebb] - thanks for looking at this. You are right - the scenario I posted
(and deleted) is impossible. The jdk docs are a little vague, but I have
confirmed you are right - the thread trying to get the write lock will have to
block until any readers release their read locks. The null guard that was
added in deregister prevents the exception reported above from happening, but
the test case that Gary added to now fails sporadically (very rarely) with this
in the stack trace:
{{java.lang.NullPointerException: Cannot invoke
"org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getIdleObjects()"
because the return value of "java.util.Map.get(Object)" is null}}
{{ at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.addIdleObject(GenericKeyedObjectPool.java:308)}}
{{ at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:333)}}
{{ at org.apache.com}}
The problem with above is that addObject calls register on the key before
invoking addIdleObject so any deregister that happens in between should not be
able to delete the pool.
The intent is for register to signal that a thread is "interested" in a key so
others should refrain from deleting it. It will also create an empty pool if
there is no pool under the given key. It returns the pool under the registered
key. Calling deregister says the thread is no longer interested and also
checks to see if the associated keyed pool is empty, has no objects under
management and has no other threads interested in it. In that case, it deletes
it. This setup ensures that a) when you call register, you can be sure that
you will get a non-null pool and that pool will not be deleted until after you
deregister it b) empty pools with no objects under management and no interested
threads get cleaned up. Thread interest is tracked in the numInterested counter
attached to the pool itself, so what must be going on is that counter must be
getting corrupted somehow. One way that could happen is if deregister is
called twice for one registration.{{{}{}}}
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
[jira] [Commented] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741106#comment-17741106
]
Phil Steitz commented on POOL-411:
--
[~sebb] - thanks for looking at this. You are right - the scenario I posted
(and deleted) is impossible. The jdk docs are a little vague, but I have
confirmed you are right - the thread trying to get the write lock will have to
block until any readers release their read locks. The null guard that was
added in deregister prevents the exception reported above from happening, but
the test case that Gary added to now fails sporadically (very rarely) with this
in the stack trace:
{{java.lang.NullPointerException: Cannot invoke
"org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getIdleObjects()"
because the return value of "java.util.Map.get(Object)" is null}}
{{ at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.addIdleObject(GenericKeyedObjectPool.java:308)}}
{{ at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:333)}}
{{ at org.apache.com}}
The problem with above is that addObject calls register on the key before
invoking addIdleObject so any deregister that happens in between should not be
able to delete the pool.
The intent is for register to signal that a thread is "interested" in a key so
others should refrain from deleting it. It will also create an empty pool if
there is no pool under the given key. It returns the pool under the registered
key. Calling deregister says the thread is no longer interested and also
checks to see if the associated keyed pool is empty, has no objects under
management and has no other threads interested in it. In that case, it deletes
it. This setup ensures that a) when you call register, you can be sure that
you will get a non-null pool and that pool will not be deleted until after you
deregister it b) empty pools with no objects under management and no interested
threads get cleaned up. Thread interest is tracked in the numInterested counter
attached to the pool itself, so what must be going on is that counter must be
getting corrupted somehow. One way that could happen is if deregister is
called twice for one registration.{{{}{}}}
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] (POOL-411) NPE when deregistering key at end of borrow
[ https://issues.apache.org/jira/browse/POOL-411 ]
Phil Steitz deleted comment on POOL-411:
--
was (Author: psteitz):
The added test case has failed at least once now. I think there is a problem
with register/deregister protection of pools. Very low probability, but the
following seems like it could happen:
Thread 1
# Enter register
# get the read lock
# get a non-null, existing pool
Thread 2
# Clear the pool
# Deregister and remove it
Thread 1
# Register interest in the now orphaned pool
This may be possible because deregister can escalate to write lock while others
are holding read locks. The Thread 2 actions have to happen between two
instructions in register (when it gets the pool and when it increments its
numInterested).
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (POOL-411) NPE when deregistering key at end of borrow
[
https://issues.apache.org/jira/browse/POOL-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz reopened POOL-411:
--
The added test case has failed at least once now. I think there is a problem
with register/deregister protection of pools. Very low probability, but the
following seems like it could happen:
Thread 1
# Enter register
# get the read lock
# get a non-null, existing pool
Thread 2
# Clear the pool
# Deregister and remove it
Thread 1
# Register interest in the now orphaned pool
This may be possible because deregister can escalate to write lock while others
are holding read locks. The Thread 2 actions have to happen between two
instructions in register (when it gets the pool and when it increments its
numInterested).
> NPE when deregistering key at end of borrow
> ---
>
> Key: POOL-411
> URL: https://issues.apache.org/jira/browse/POOL-411
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.11.1
>Reporter: Richard Eckart de Castilho
>Priority: Major
> Fix For: 2.12.0
>
>
> There is a potential for an NPE happening in the finally block of
> borrowObject:
> {noformat}
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.commons.pool2.impl.GenericKeyedObjectPool$ObjectDeque.getNumInterested()"
> because "objectDeque" is null
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.deregister(GenericKeyedObjectPool.java:821)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:507)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
>
> {noformat}
> From reading the code, it seems this could happen e.g. if a pool is
> concurrently cleared while a borrow is in progress.
> Not sure what a proper solution here would be. Maybe deregister should
> silently do nothing if poolMap.get(k) returns null?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DBCP-591) Avoid synchronized in PoolableConnection.close
[ https://issues.apache.org/jira/browse/DBCP-591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738305#comment-17738305 ] Phil Steitz commented on DBCP-591: -- Acknowledged. It is strange that intrinsic locks work differently, but I understand that at this point at least they do. I am tempted to suggest removing the sync entirely here; but we have not been that consistent in stating the expectation that concurrent access to PoolableConnections is not a good thing, so it is probably best to leave it in, but replace the intrinsic lock with an a Reeentrant lock. Patches / better ideas welcome. > Avoid synchronized in PoolableConnection.close > -- > > Key: DBCP-591 > URL: https://issues.apache.org/jira/browse/DBCP-591 > Project: Commons DBCP > Issue Type: Improvement >Reporter: Attila Kelemen >Priority: Minor > > org.apache.commons.dbcp2.PoolableConnection.close uses an intrinsic lock to > synchronize its close method. However, this can be an issue since this pins > the carrier thread of a virtual thread, needlessly degrading performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IO-802) Restore threadlocal for skipfully() byte buffer
[
https://issues.apache.org/jira/browse/IO-802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17732273#comment-17732273
]
Phil Steitz commented on IO-802:
[~ggregory] Makes sense. Thanks.
> Restore threadlocal for skipfully() byte buffer
> ---
>
> Key: IO-802
> URL: https://issues.apache.org/jira/browse/IO-802
> Project: Commons IO
> Issue Type: Bug
>Reporter: Tim Allison
>Priority: Major
>
> Over on TIKA-4065, we found that trying to upgrade to commons-io 2.12.0 or
> 2.13.0 caused one of our unit tests to fail. We found that dropping
> {{threadlocal}} on the buffer used in IOUtils.skipFully() in conjunction with
> Java's InflaterInputStream was the cause of the problem.
> Our unit test shows that running skipFully() on a stream and then reading
> gets different results on the same underlying stream when running
> multithreaded. This is really bad. It appears to be confined to
> InflaterInputStream...so not a very common case.
> On the [commons-io's user
> list|https://lists.apache.org/thread/rxfyxqochnj7bw75nr2v7hf5qtkogx7d]
> [~psteitz] observed that Java's InflaterInputStream expects read access to
> the byte array passed in...so having multiple threads writing to the same
> static (not-thread local) byte array is dangerous. The behavior of Java's
> InflaterInputStream is surprising and not documented.
> I have a demonstration of the problem here:
> https://github.com/tballison/commons-io/blob/TIKA-4065/src/test/java/org/apache/commons/io/IOUtilsMultithreadedTest.java
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IO-802) Restore threadlocal for skipfully() byte buffer
[
https://issues.apache.org/jira/browse/IO-802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17732238#comment-17732238
]
Phil Steitz commented on IO-802:
I agree that it would be best to just add back the protection, for the reasons
above. The jdk developers at least in the InflaterInputStream case seem to
think concurrent writes to a read buffer should not happen, so I would
recommend eliminating this risk. I would also evaluate the need for zero-ing
for the write only one. Is it needed?
> Restore threadlocal for skipfully() byte buffer
> ---
>
> Key: IO-802
> URL: https://issues.apache.org/jira/browse/IO-802
> Project: Commons IO
> Issue Type: Bug
>Reporter: Tim Allison
>Priority: Major
>
> Over on TIKA-4065, we found that trying to upgrade to commons-io 2.12.0 or
> 2.13.0 caused one of our unit tests to fail. We found that dropping
> {{threadlocal}} on the buffer used in IOUtils.skipFully() in conjunction with
> Java's InflaterInputStream was the cause of the problem.
> Our unit test shows that running skipFully() on a stream and then reading
> gets different results on the same underlying stream when running
> multithreaded. This is really bad. It appears to be confined to
> InflaterInputStream...so not a very common case.
> On the [commons-io's user
> list|https://lists.apache.org/thread/rxfyxqochnj7bw75nr2v7hf5qtkogx7d]
> [~psteitz] observed that Java's InflaterInputStream expects read access to
> the byte array passed in...so having multiple threads writing to the same
> static (not-thread local) byte array is dangerous. The behavior of Java's
> InflaterInputStream is surprising and not documented.
> I have a demonstration of the problem here:
> https://github.com/tballison/commons-io/blob/TIKA-4065/src/test/java/org/apache/commons/io/IOUtilsMultithreadedTest.java
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-412) [GenericKeyedObjectPool] ensureMinIdle not work if last idle evicted
[
https://issues.apache.org/jira/browse/POOL-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729462#comment-17729462
]
Phil Steitz commented on POOL-412:
--
Yes, I would close this.
> [GenericKeyedObjectPool] ensureMinIdle not work if last idle evicted
>
>
> Key: POOL-412
> URL: https://issues.apache.org/jira/browse/POOL-412
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.11.1
>Reporter: Oleksandr Porytskyi
>Priority: Major
>
> I'm trying to use GenericKeyedObjectPool with setMinIdlePerKey(1) and
> setTestWhileIdle(true). When object failed validation it is removed but
> never add new one.
> Evictor has two stages:
> 1. In evict() call destroy() -> deregister(key) -> poolMap.remove(k)
> For some reason it removes key if there are no more objects of it.
>
> 2. In ensureMinIdle() ->
> for (final K k : poolMap.keySet()) {
> ensureMinIdle(k);
> }
> poolMap does not have my key anymore so will not create object for it.
>
> Here one test for GenericObjectPool which pass and one for
> GenericKeyedObjectPool which not pass for same scenario:
> {code:java}
> @Test
> void testGenericKeyedObjectPool() throws Exception {
> BaseKeyedPooledObjectFactory baseKeyedPooledObjectFactory =
> new BaseKeyedPooledObjectFactory<>() {
> @Override
> public Object create(String key) throws Exception {
> return null;
> }
> @Override
> public PooledObject wrap(Object value) {
> return new DefaultPooledObject<>(value);
> }
> };
> GenericKeyedObjectPoolConfig genericKeyedObjectPoolConfig = new
> GenericKeyedObjectPoolConfig<>();
> int minIdlePerKey = 1;
> genericKeyedObjectPoolConfig.setMinIdlePerKey(minIdlePerKey);
>
> genericKeyedObjectPoolConfig.setTimeBetweenEvictionRuns(Duration.ofSeconds(1));
> genericKeyedObjectPoolConfig.setMinEvictableIdleTime(Duration.ofSeconds(5));
> GenericKeyedObjectPool genericKeyedObjectPool = new
> GenericKeyedObjectPool<>(
> baseKeyedPooledObjectFactory, genericKeyedObjectPoolConfig);
> String key = "key";
> genericKeyedObjectPool.preparePool(key);
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while (genericKeyedObjectPool.getNumIdle(key) != minIdlePerKey)
> ;
> });
> System.out.println("we prepared pool so we have idle");
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while (genericKeyedObjectPool.getNumIdle(key) != 0)
> ;
> });
> System.out.println("after eviction we have no idle");
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while (genericKeyedObjectPool.getNumIdle(key) != minIdlePerKey)
> ;
> });
> System.out.println("NEVER HAPPEN: after eviction ensure min idle");
> }
> @Test
> void testGenericObjectPool() throws Exception {
> BasePooledObjectFactory basePooledObjectFactory = new
> BasePooledObjectFactory<>() {
> @Override
> public Object create() throws Exception {
> return null;
> }
> @Override
> public PooledObject wrap(Object obj) {
> return new DefaultPooledObject<>(obj);
> }
> };
> GenericObjectPoolConfig genericObjectPoolConfig = new
> GenericObjectPoolConfig<>();
> int minIdle = 1;
> genericObjectPoolConfig.setTimeBetweenEvictionRuns(Duration.ofSeconds(1));
> genericObjectPoolConfig.setMinIdle(minIdle);
> genericObjectPoolConfig.setMinEvictableIdleTime(Duration.ofSeconds(5));
> GenericObjectPool genericObjectPool = new
> GenericObjectPool<>(basePooledObjectFactory,
> genericObjectPoolConfig);
> genericObjectPool.preparePool();
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while (genericObjectPool.getNumIdle() != minIdle)
> ;
> });
> System.out.println("we prepared pool so we have idle");
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while (genericObjectPool.getNumIdle() != 0)
> ;
> });
> System.out.println("after eviction we have no idle");
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while (genericObjectPool.getNumIdle() != minIdle)
> ;
> });
> System.out.println("after eviction ensure min idle");
> }
> {code}
> As workaround I can't just subclass GenericKeyedObjectPool and alter
> deregister as it private.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-412) [GenericKeyedObjectPool] ensureMinIdle not work if last idle evicted
[
https://issues.apache.org/jira/browse/POOL-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729430#comment-17729430
]
Phil Steitz commented on POOL-412:
--
This is an interesting use case. Honestly, I think GKOP is working as designed
here. There is no contract to maintain an immutable, durable set of keys.
Once the last object instance under a key is destroyed, that key no longer
exists and ensureMinIdle can't be expected to replenish instances under the key
that is no longer there. Admittedly, eviction as the cause of key death is a
little extreme, but I can imagine use cases where that is the desired behavior.
To "fix" this would require that we keep the key alive which for the OPs use
case is desired, but may not be desired in other cases and would have to be
extended to failed validations, abandoned objects and all other cases where
keyed pools are exhausted. As a workaround for the OP, I would recommend
using addObject to restore the keyed pool when it has been removed. I would
not recommend changing the GKOP contract to support durable keys.
> [GenericKeyedObjectPool] ensureMinIdle not work if last idle evicted
>
>
> Key: POOL-412
> URL: https://issues.apache.org/jira/browse/POOL-412
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.11.1
>Reporter: Oleksandr Porytskyi
>Priority: Major
>
> I'm trying to use GenericKeyedObjectPool with setMinIdlePerKey(1) and
> setTestWhileIdle(true). When object failed validation it is removed but
> never add new one.
> Evictor has two stages:
> 1. In evict() call destroy() -> deregister(key) -> poolMap.remove(k)
> For some reason it removes key if there are no more objects of it.
>
> 2. In ensureMinIdle() ->
> for (final K k : poolMap.keySet()) {
> ensureMinIdle(k);
> }
> poolMap does not have my key anymore so will not create object for it.
>
> Here one test for GenericObjectPool which pass and one for
> GenericKeyedObjectPool which not pass for same scenario:
> {code:java}
> @Test
> void testGenericKeyedObjectPool() throws Exception {
> BaseKeyedPooledObjectFactory baseKeyedPooledObjectFactory =
> new BaseKeyedPooledObjectFactory<>() {
> @Override
> public Object create(String key) throws Exception {
> return null;
> }
> @Override
> public PooledObject wrap(Object value) {
> return new DefaultPooledObject<>(value);
> }
> };
> GenericKeyedObjectPoolConfig genericKeyedObjectPoolConfig = new
> GenericKeyedObjectPoolConfig<>();
> int minIdlePerKey = 1;
> genericKeyedObjectPoolConfig.setMinIdlePerKey(minIdlePerKey);
>
> genericKeyedObjectPoolConfig.setTimeBetweenEvictionRuns(Duration.ofSeconds(1));
> genericKeyedObjectPoolConfig.setMinEvictableIdleTime(Duration.ofSeconds(5));
> GenericKeyedObjectPool genericKeyedObjectPool = new
> GenericKeyedObjectPool<>(
> baseKeyedPooledObjectFactory, genericKeyedObjectPoolConfig);
> String key = "key";
> genericKeyedObjectPool.preparePool(key);
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while (genericKeyedObjectPool.getNumIdle(key) != minIdlePerKey)
> ;
> });
> System.out.println("we prepared pool so we have idle");
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while (genericKeyedObjectPool.getNumIdle(key) != 0)
> ;
> });
> System.out.println("after eviction we have no idle");
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while (genericKeyedObjectPool.getNumIdle(key) != minIdlePerKey)
> ;
> });
> System.out.println("NEVER HAPPEN: after eviction ensure min idle");
> }
> @Test
> void testGenericObjectPool() throws Exception {
> BasePooledObjectFactory basePooledObjectFactory = new
> BasePooledObjectFactory<>() {
> @Override
> public Object create() throws Exception {
> return null;
> }
> @Override
> public PooledObject wrap(Object obj) {
> return new DefaultPooledObject<>(obj);
> }
> };
> GenericObjectPoolConfig genericObjectPoolConfig = new
> GenericObjectPoolConfig<>();
> int minIdle = 1;
> genericObjectPoolConfig.setTimeBetweenEvictionRuns(Duration.ofSeconds(1));
> genericObjectPoolConfig.setMinIdle(minIdle);
> genericObjectPoolConfig.setMinEvictableIdleTime(Duration.ofSeconds(5));
> GenericObjectPool genericObjectPool = new
> GenericObjectPool<>(basePooledObjectFactory,
> genericObjectPoolConfig);
> genericObjectPool.preparePool();
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while (genericObjectPool.getNumIdle() != minIdle)
> ;
> });
> System.out.println("we prepared pool so we have idle");
> Assertions.assertTimeoutPreemptively(Duration.ofMinutes(1), () -> {
> while
[jira] [Comment Edited] (DBCP-590) BasicDataSource#setAbandonedUsageTracking has no effect
[
https://issues.apache.org/jira/browse/DBCP-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681805#comment-17681805
]
Phil Steitz edited comment on DBCP-590 at 1/30/23 2:18 AM:
---
I will look more at this when I have a little more time. IIRC, this
functionality used to work. I would not recommend changing the datasource
creation until we understand what is causing the problem. Could be I am wrong
and it did not work before. I will do some testing and analysis some time this
week.
was (Author: psteitz):
I will look more at this when I have a little more time, but I suspect this may
be a regression due to the refactoring changes in
344234a758da62aa422f6f67c6e7b1f37329c9c0. IIRC, this functionality used to
work. I would not recommend changing the datasource creation until we
understand what is causing the problem. Could be I am wrong and it did not
work before. I will do some testing and analysis some time this week.
> BasicDataSource#setAbandonedUsageTracking has no effect
> ---
>
> Key: DBCP-590
> URL: https://issues.apache.org/jira/browse/DBCP-590
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Réda Housni Alaoui
>Priority: Major
>
> Passing {{true}} to {{BasicDataSource#setAbandonedUsageTracking(boolean
> usageTracking)}} has no effect because {{UsageTracking#use}} is never called.
> From what I found, {{usageTracking}} can only work if the object pool is of
> type {{ProxiedObjectPool}} . Alas, BasicDataSource enforces
> {{GenericObjectPool}} concrete type preventing us from overriding
> {{BasicDataSource#createObjectPool}} to return a {{ProxiedObjectPool}} .
> Is there something I missed or a workaround?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DBCP-590) BasicDataSource#setAbandonedUsageTracking has no effect
[
https://issues.apache.org/jira/browse/DBCP-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681805#comment-17681805
]
Phil Steitz commented on DBCP-590:
--
I will look more at this when I have a little more time, but I suspect this may
be a regression due to the refactoring changes in
344234a758da62aa422f6f67c6e7b1f37329c9c0. IIRC, this functionality used to
work. I would not recommend changing the datasource creation until we
understand what is causing the problem. Could be I am wrong and it did not
work before. I will do some testing and analysis some time this week.
> BasicDataSource#setAbandonedUsageTracking has no effect
> ---
>
> Key: DBCP-590
> URL: https://issues.apache.org/jira/browse/DBCP-590
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Réda Housni Alaoui
>Priority: Major
>
> Passing {{true}} to {{BasicDataSource#setAbandonedUsageTracking(boolean
> usageTracking)}} has no effect because {{UsageTracking#use}} is never called.
> From what I found, {{usageTracking}} can only work if the object pool is of
> type {{ProxiedObjectPool}} . Alas, BasicDataSource enforces
> {{GenericObjectPool}} concrete type preventing us from overriding
> {{BasicDataSource#createObjectPool}} to return a {{ProxiedObjectPool}} .
> Is there something I missed or a workaround?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-401) GKOP invalidateObject should make capacity available to all keyed pools
[ https://issues.apache.org/jira/browse/POOL-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653706#comment-17653706 ] Phil Steitz commented on POOL-401: -- It looks to me that at least as of 2.11.1 reuseCapacity is called on each activation of invalidateObject in GKOP, so if the freed capacity should be added to the most heavily loaded pool. Could be I am missing something. A unit test showing starvation would help. > GKOP invalidateObject should make capacity available to all keyed pools > --- > > Key: POOL-401 > URL: https://issues.apache.org/jira/browse/POOL-401 > Project: Commons Pool > Issue Type: Bug >Reporter: Phil Steitz >Priority: Major > > GKOP invalidateObject currently only creates and adds objects if there are > borrowers waiting for the keyed pool that the instance being invalidated came > from. If there are no take waiters on that pool, or there is a more heavily > loaded pool, the (total) capacity should be reused in a different pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (POOL-407) Threads get stuck when idleObjects list is empty.
[
https://issues.apache.org/jira/browse/POOL-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653705#comment-17653705
]
Phil Steitz commented on POOL-407:
--
Thanks for reporting this with the detailed analysis. It is a liveness bug.
One option to consider for a patch would be to modify the code in borrowObject
to do what returnObject does on validation failures, which is to call
ensureIdle(1, false). Need to think carefully about the consequences of adding
this, but that is what we did to address the dual of this problem on return.
> Threads get stuck when idleObjects list is empty.
> -
>
> Key: POOL-407
> URL: https://issues.apache.org/jira/browse/POOL-407
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Sarthak Shukla
>Priority: Major
>
> While borrowing object from pool, threads are getting stuck. I initialised
> the pool size as 1. And had 3 threads created. First thread enters
> borrowObject method, since there are no idle objects to poll from, it will
> create one object and move forward.
> {code:java}
> p = (PooledObject)this.idleObjects.pollFirst();
> if (p == null) {
> p = this.create();
> if (p != null) {
> create = true;
> }
> } {code}
> The other two threads will also follow same path and check for idle
> objects(there are none), will try to create one object but the pool size is
> set to 1. Thus, the two threads will move forward and enter
> *idleObjects.takeFirst()* function. Value of blockWhenExhausted is true and
> borrowMaxWaitMillis is -1 as we don't want timeout.
> {code:java}
> if (blockWhenExhausted) {
>if (p == null) {
> if (borrowMaxWaitMillis < 0L) {
>p = (PooledObject)this.idleObjects.takeFirst();
> } else {
>p = (PooledObject)this.idleObjects.pollFirst(borrowMaxWaitMillis,
> TimeUnit.MILLISECONDS);
> }
>}
>if (p == null) {
> throw new NoSuchElementException("Timeout waiting for idle object");
>}
> }{code}
> Now, the main thread does *this.factory.activateObject(p);* and object gets
> activated. Now, when the validation is checked *validate =
> this.factory.validateObject(p);* it comes out to be false as provider might
> have been disconnected.
> So, the object is destroyed by calling *this.destroy(p);*
> {code:java}
> private void destroy(PooledObject toDestroy) throws Exception {
> toDestroy.invalidate();
> this.idleObjects.remove(toDestroy);
> this.allObjects.remove(new
> BaseGenericObjectPool.IdentityWrapper(toDestroy.getObject()));
> try {
> this.factory.destroyObject(toDestroy);
> } finally {
> this.destroyedCount.incrementAndGet();
> this.createCount.decrementAndGet();
> }
> }{code}
> The object which was created is now destroyed and removed from idleObject and
> allObjects list. Now, the other two threads are still waiting to take object
> from idle objects list but there are no object present. Hence, the two
> threads are in wait state for infinite period and the application waits
> forever until we kill the process.
> {code:java}
> public E takeFirst() throws InterruptedException {
>this.lock.lock();
>Object var2;
>try {
> Object x;
> while((x = this.unlinkFirst()) == null) {
> this.notEmpty.await();
> }
> var2 = x;
> } finally {
> this.lock.unlock();
> }
> return var2;
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-350) Add option for not executing "hasBorrowWaiters()" while returning objects
[
https://issues.apache.org/jira/browse/POOL-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653704#comment-17653704
]
Phil Steitz commented on POOL-350:
--
Any feedback on the reuseCapacityOnReturn, reuseCapacityOnMaintenance idea?
Might help in situations like the OP's. Should not be hard to implement.
> Add option for not executing "hasBorrowWaiters()" while returning objects
> -
>
> Key: POOL-350
> URL: https://issues.apache.org/jira/browse/POOL-350
> Project: Commons Pool
> Issue Type: New Feature
>Affects Versions: 2.6.0
> Environment: h5. uname -a:
> Linux VMS26239 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015
> x86_64 x86_64 x86_64 GNU/Linux
>
> *Java version:*
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>
>
>Reporter: zhu chen
>Priority: Critical
> Labels: easyfix
>
> h2. Phenomena:
> I'm recently leveraging commons-pool as my Redis connection pool in my
> project, however, the pain is that when my system is dealing with over
> thousands of Redises, CPU load become such high. By checking JVM through
> JFR(FlightRecorder), it turned out the hot method was
> "{color:#FF}hasBorrowWaiters(){color}", which is invoked by
> "{color:#FF}returnObject(){color}" each time.
> That means the system will go through over *thousands*(the number will grow
> as well as my system) of keys after *each* object's *return*, what's worse,
> the program is running concurrently, which, obviously cause a huge CPU load.
>
> h2. Expect:
> I was wondering if we could add a config for optionally run this
> "hasBorrowWaiters()" each time when we return an object.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (POOL-372) CallStackUtils mishandles security manager check part 2
[ https://issues.apache.org/jira/browse/POOL-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653703#comment-17653703 ] Phil Steitz commented on POOL-372: -- I am not a security expert, so I can't comment on the assertion in (b) above, but at least as of 2.10, (a) does not apply. I like the idea of replacing the impl using Flight recorder though, if we can figure out a way to do it. The reason that this exists is to allow pooled objects to hold onto the stack trace leading to their creation. What is not obvious to me about how to use Flight Recorder is knowing what to capture in advance. > CallStackUtils mishandles security manager check part 2 > --- > > Key: POOL-372 > URL: https://issues.apache.org/jira/browse/POOL-372 > Project: Commons Pool > Issue Type: Bug >Reporter: Volker Kleinschmidt >Priority: Major > > This ticket is for (b). > CallStackUtils determines at initialization time whether it is allowed to > create a security manager, then sticks that info into a static variable and > never checks it again, relying on this check to later try to create a > SecurityManager for a SecurityManagerCallStack. This is doubly wrong: > a) If the code is running in a privileged context at init time, it determines > that it can create a security manager, and then later naively assumes that > henceforth all code is privileged and also can create a security manager. Of > course this is not true, otherwise one would not need a security manager in > the first place! This info can never be kept in a static variable, it's > extremely context-dependent. So this leads to AccessControlException from > invoking newCallStack() if abandoned object logging is enabled. > b) The permission to create a security manager must never be granted to any > code, unless that code has AllPermission in the first place, i.e. is already > fully privileged. This is because this permission allows circumventing the > security manager completely (simply create one that lets all checks pass). > Therefore even just checking whether you're allowed to create a secmgr is > naive - if a secmgr is installed at all you should assume that you're NOT > privileged enough to do this, simply because for sure some code that calls > your code will not be privileged enough. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (POOL-264) NullPointerException in GKOP.borrowObject()
[ https://issues.apache.org/jira/browse/POOL-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653699#comment-17653699 ] Phil Steitz commented on POOL-264: -- The entire setup was changed in 2.0. Unless someone wants to take up the task of creating a 1.6.x, I think we should close this as WONT_FIX. Users of 1.x versions of commons pool should upgrade to version 2+ > NullPointerException in GKOP.borrowObject() > --- > > Key: POOL-264 > URL: https://issues.apache.org/jira/browse/POOL-264 > Project: Commons Pool > Issue Type: Bug >Affects Versions: 1.5.6, 1.5.7, 1.6 >Reporter: Leonid Meyerguz >Priority: Major > > While I cannot pin down a consistent repro, I occasionally observe a > NullPointerException at > org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1126) > The pool is configured as follows: > maxActive = -1 > maxIdle = 32 > maxTotal = 32 > whenExhaustedAction = WHEN_EXHAUSTED_GROW > timeBetweenEvictionRunsMillis = 2 > minEvictableIdleTimeMillis = 6 > numTestsPerEvictionRun = -1 > The NullPointerException is thrown in the WHEN_EXHAUSTED_GROW branch of the > code. Specifically it appears that latch.getPool() returns null. > Any suggestions for a work-around would be appreciated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DBCP-587) DBCP and Transparent Application Continuity
[ https://issues.apache.org/jira/browse/DBCP-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622847#comment-17622847 ] Phil Steitz commented on DBCP-587: -- Thanks. It was a good question. > DBCP and Transparent Application Continuity > --- > > Key: DBCP-587 > URL: https://issues.apache.org/jira/browse/DBCP-587 > Project: Commons DBCP > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Kirk Hill >Priority: Major > > Oracle databases have a high-availability setup that uses an item called > Transparent Application Continuity. It requires using the following driver > class name for "oracle.jdbc.replay.OracleDataSourceImpl" When I attempt to > use this driver I get the following error message. > SQLException occurred : Cannot create JDBC driver of class > 'oracle.jdbc.replay.OracleDataSourceImpl' > Having this as a way to create connection pools would greatly enhance your > product. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DBCP-587) DBCP and Transparent Application Continuity
[ https://issues.apache.org/jira/browse/DBCP-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17619934#comment-17619934 ] Phil Steitz commented on DBCP-587: -- GenericObjectPool has no initialSize parameter. You need to do what BasicDataSource does to pre-load the pool, which is to call objectPool.addObject initialSize times to get the initial instances created. Alternatively, since you have set minIdle, you can call GOP's preparePool method which will create 4 instances. It would be great to either summarize the discussion here or post a link back to your question on commons-user. > DBCP and Transparent Application Continuity > --- > > Key: DBCP-587 > URL: https://issues.apache.org/jira/browse/DBCP-587 > Project: Commons DBCP > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Kirk Hill >Priority: Major > > Oracle databases have a high-availability setup that uses an item called > Transparent Application Continuity. It requires using the following driver > class name for "oracle.jdbc.replay.OracleDataSourceImpl" When I attempt to > use this driver I get the following error message. > SQLException occurred : Cannot create JDBC driver of class > 'oracle.jdbc.replay.OracleDataSourceImpl' > Having this as a way to create connection pools would greatly enhance your > product. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (POOL-282) CLONE - Support AbandonedConfig in SharedPoolDataSource
[ https://issues.apache.org/jira/browse/POOL-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531339#comment-17531339 ] Phil Steitz edited comment on POOL-282 at 5/3/22 6:08 PM: -- [~ashok2ashok] First, many thanks for stepping up to the challenge here. Here are some comments: # I don't think we need to support different abandoned configs for different keys. That would be kind of messy and I can't immediately think of scenarios where different configs would be desired. Does your use case require this? Assuming no, I would stick with just using the AbandonedConfig class as is. # The per key considerations come in when you actually do the cleanup. It seems reasonable to me to generally follow the pattern of what GOP does, except that you add the key as a parameter to removeAbandoned, so it is ``removeAbandoned(ac, key)`` and it does what the signature suggests - just goes after abandoned instances under the given key. So borrowObject does the same kind of test that GOP does and if the pool is running low for the given key, it does the cleanup for that key. # Maintenance is kind of interesting. The natural thing to do would be to iterate all keys and call ``removeAbandoned(ac, key)`` for all of them. For pools with lots of keys, that might cause performance problems. I can see three options here: 0) don't worry about it 1) hitchhike on the iteration sequence in the loop over numTests in evict and just hit the keys that get visited 2) introduce a separate config that controls how many keys get cleaned up per eviction run and maintain a separate keys iterator to cycle through them. Could be I am needlessly worrying here and 0) is fine. The whole thing can be turned off by setting removeAbandonedOnMaintenance to false so I think it is probably OK to start with the simple hit them all impl. was (Author: psteitz): [~ashok2ashok] First, many thanks for stepping up to the challenge here. Here are some comments: # I don't think we need to support different abandoned configs for different keys. That would be kind of messy and I can't immediately think of scenarios where different configs would be desired. Does your use case require this? Assuming no, I would stick with just using the AbandonedConfig class as is. # The per key considerations come in when you actually do the cleanup. It seems reasonable to me to generally follow the pattern of what GOP does, except that you add the key as a parameter to removeAbandoned, so it is `removeAbandoned(ac, key)` and it does what the signature suggests - just goes after abandoned instances under the given key. So borrowObject does the same kind of test that GOP does and if the pool is running low for the given key, it does the cleanup for that key. # Maintenance is kind of interesting. The natural thing to do would be to iterate all keys and call `removeAbandoned(ac, key)` for all of them. For pools with lots of keys, that might cause performance problems. I can see three options here: 0) don't worry about it 1) hitchhike on the iteration sequence in the loop over numTests in evict and just hit the keys that get visited 2) introduce a separate config that controls how many keys get cleaned up per eviction run and maintain a separate keys iterator to cycle through them. Could be I am needlessly worrying here and 0) is fine. The whole thing can be turned off by setting removeAbandonedOnMaintenance to false so I think it is probably OK to start with the simple hit them all impl. > CLONE - Support AbandonedConfig in SharedPoolDataSource > --- > > Key: POOL-282 > URL: https://issues.apache.org/jira/browse/POOL-282 > Project: Commons Pool > Issue Type: Improvement >Affects Versions: 2.0, 2.1, 2.2, 2.3 > Environment: Linux and Java 1.6 >Reporter: Michael Kerr >Priority: Minor > Labels: features > > BasicDataSource has support for AbandonedConfig > SharedPoolDataSource does not currently support reclaiming abandoned > connections. > It would be helpful if the user can extend the underlying connection pool to > add features like abandoned connection management. This is possible with > BasicDataSource. > Currently the only solution is to extend classes or replace classes in the > Apache package namespace. > Thanks for looking, > Michael -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (POOL-282) CLONE - Support AbandonedConfig in SharedPoolDataSource
[ https://issues.apache.org/jira/browse/POOL-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531339#comment-17531339 ] Phil Steitz commented on POOL-282: -- [~ashok2ashok] First, many thanks for stepping up to the challenge here. Here are some comments: # I don't think we need to support different abandoned configs for different keys. That would be kind of messy and I can't immediately think of scenarios where different configs would be desired. Does your use case require this? Assuming no, I would stick with just using the AbandonedConfig class as is. # The per key considerations come in when you actually do the cleanup. It seems reasonable to me to generally follow the pattern of what GOP does, except that you add the key as a parameter to removeAbandoned, so it is `removeAbandoned(ac, key)` and it does what the signature suggests - just goes after abandoned instances under the given key. So borrowObject does the same kind of test that GOP does and if the pool is running low for the given key, it does the cleanup for that key. # Maintenance is kind of interesting. The natural thing to do would be to iterate all keys and call `removeAbandoned(ac, key)` for all of them. For pools with lots of keys, that might cause performance problems. I can see three options here: 0) don't worry about it 1) hitchhike on the iteration sequence in the loop over numTests in evict and just hit the keys that get visited 2) introduce a separate config that controls how many keys get cleaned up per eviction run and maintain a separate keys iterator to cycle through them. Could be I am needlessly worrying here and 0) is fine. The whole thing can be turned off by setting removeAbandonedOnMaintenance to false so I think it is probably OK to start with the simple hit them all impl. > CLONE - Support AbandonedConfig in SharedPoolDataSource > --- > > Key: POOL-282 > URL: https://issues.apache.org/jira/browse/POOL-282 > Project: Commons Pool > Issue Type: Improvement >Affects Versions: 2.0, 2.1, 2.2, 2.3 > Environment: Linux and Java 1.6 >Reporter: Michael Kerr >Priority: Minor > Labels: features > > BasicDataSource has support for AbandonedConfig > SharedPoolDataSource does not currently support reclaiming abandoned > connections. > It would be helpful if the user can extend the underlying connection pool to > add features like abandoned connection management. This is possible with > BasicDataSource. > Currently the only solution is to extend classes or replace classes in the > Apache package namespace. > Thanks for looking, > Michael -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (DBCP-585) Connection level JMX queries result in concurrent access to connection objects, causing errors
[
https://issues.apache.org/jira/browse/DBCP-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530947#comment-17530947
]
Phil Steitz commented on DBCP-585:
--
[~ggregory] yes this looks good
> Connection level JMX queries result in concurrent access to connection
> objects, causing errors
> --
>
> Key: DBCP-585
> URL: https://issues.apache.org/jira/browse/DBCP-585
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Kurtcebe Eroglu
>Priority: Major
> Attachments: 0001-DBCP-585-idea-clarification-1.patch,
> conn_instance_attrs.png, connections_attrs.png, ds_attrs.png, final.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> As we expose Connection objects over JMX, they may be accessed by multiple
> threads concurrently;
> a) an application thread that borrows the Connection and uses it business as
> usual,
> b) another thread simultaneously performing a JMX query, which in turn calls
> getters on the same connection object via the MBean interface.
> Also, calls to Connection object getters are mostly delegated to the
> underlying vendor-specific connection provided by the JDBC driver. For
> example, when we make the JMX query to get the "schema" attribute of the JMX
> connection object, this is translated into a
> "java.sql.Connection.getSchema()", and passed to the vendor-specific
> Connection object by DBCP. In the case of Postgres, for example, this is
> further translated to a query "select current_schema()" and sent to the
> server.
> Hence, querying connections over JMX result in concurrent access by multiple
> threads to the underlying Connection provided by the vendors, to the point
> that these two threads may be running queries simultaneously on the same
> connection.
> However, this is not supported by any of the major database vendors. Vendor
> links on Connection objects not being threadsafe:
> - [Postgres|https://jdbc.postgresql.org/documentation/head/thread.html]
> {quote}The PostgreSQL™ JDBC driver is not thread safe. The PostgreSQL server
> is not threaded. Each connection creates a new process on the server; as such
> any concurrent requests to the process would have to be serialized. The
> driver makes no guarantees that methods on connections are synchronized. It
> will be up to the caller to synchronize calls to the driver.
> {quote}
> -
> [Oracle|https://docs.oracle.com/en/database/oracle/oracle-database/19/jjdbc/JDBC-coding-tips.html#GUID-EE479007-D105-4F82-8D51-000CBBD4BC77]
>
> {quote}Oracle strongly discourages sharing a database connection among
> multiple threads. Avoid allowing multiple threads to access a connection
> simultaneously.
> {quote}
> - [Microsoft SQL
> Server|https://www.javadoc.io/doc/com.microsoft.sqlserver/mssql-jdbc/latest/com.microsoft.sqlserver.jdbc/com/microsoft/sqlserver/jdbc/SQLServerConnection.html]
> {quote}SQLServerConnection is not thread safe, however multiple statements
> created from a single connection can be processing simultaneously in
> concurrent threads.
> {quote}
> Another interesting point to note, also to do justice to previous committers
> who have put this feature in place, is that this was not always the case. In
> the following links, you may see the same links to the older versions of the
> same pages. In the past, all vendors indicated that Connection is fully
> thread-safe; [Postgres|https://www.postgresql.org/docs/7.1/jdbc-thread.html],
> [Oracle|https://docs.oracle.com/cd/A97335_02/apps.102/a83724/tips1.htm],
> [MSSQL
> Server|https://www.javadoc.io/doc/com.microsoft.sqlserver/mssql-jdbc/6.1.0.jre7/com/microsoft/sqlserver/jdbc/SQLServerConnection.html].
>
> Hence, it was once safe to expose Connection objects via JMX given the
> thread-safety guarantees for the underlying vendor connection were in place.
> But as Vendors dropped the thread-safety guarantee one by one, it is not safe
> anymore, and may actually cause convoluted errors that pop up intermittently
> due to thread races in the JDBC driver code (see an example in the comments
> section below). Accordingly, exposing Connections via JMX shall be retired
> along with dropped support from most vendors.
> Note: the Datasource MBeans, which provide a vital set of metrics have no
> such problems as they don't depend on the underlying JDBC provider.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (DBCP-585) Connection level JMX queries result in concurrent access to connection objects, causing errors
[
https://issues.apache.org/jira/browse/DBCP-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17526928#comment-17526928
]
Phil Steitz commented on DBCP-585:
--
[~ggregory] sorry I missed this. There is a small javadoc error in the patch
(says "sets t." when it means something else). There should also be some tests
added to verify it is working as designed. But in general, yes, I am +1 on
this.
> Connection level JMX queries result in concurrent access to connection
> objects, causing errors
> --
>
> Key: DBCP-585
> URL: https://issues.apache.org/jira/browse/DBCP-585
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Kurtcebe Eroglu
>Priority: Major
> Attachments: 0001-DBCP-585-idea-clarification-1.patch,
> conn_instance_attrs.png, connections_attrs.png, ds_attrs.png, final.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> As we expose Connection objects over JMX, they may be accessed by multiple
> threads concurrently;
> a) an application thread that borrows the Connection and uses it business as
> usual,
> b) another thread simultaneously performing a JMX query, which in turn calls
> getters on the same connection object via the MBean interface.
> Also, calls to Connection object getters are mostly delegated to the
> underlying vendor-specific connection provided by the JDBC driver. For
> example, when we make the JMX query to get the "schema" attribute of the JMX
> connection object, this is translated into a
> "java.sql.Connection.getSchema()", and passed to the vendor-specific
> Connection object by DBCP. In the case of Postgres, for example, this is
> further translated to a query "select current_schema()" and sent to the
> server.
> Hence, querying connections over JMX result in concurrent access by multiple
> threads to the underlying Connection provided by the vendors, to the point
> that these two threads may be running queries simultaneously on the same
> connection.
> However, this is not supported by any of the major database vendors. Vendor
> links on Connection objects not being threadsafe:
> - [Postgres|https://jdbc.postgresql.org/documentation/head/thread.html]
> {quote}The PostgreSQL™ JDBC driver is not thread safe. The PostgreSQL server
> is not threaded. Each connection creates a new process on the server; as such
> any concurrent requests to the process would have to be serialized. The
> driver makes no guarantees that methods on connections are synchronized. It
> will be up to the caller to synchronize calls to the driver.
> {quote}
> -
> [Oracle|https://docs.oracle.com/en/database/oracle/oracle-database/19/jjdbc/JDBC-coding-tips.html#GUID-EE479007-D105-4F82-8D51-000CBBD4BC77]
>
> {quote}Oracle strongly discourages sharing a database connection among
> multiple threads. Avoid allowing multiple threads to access a connection
> simultaneously.
> {quote}
> - [Microsoft SQL
> Server|https://www.javadoc.io/doc/com.microsoft.sqlserver/mssql-jdbc/latest/com.microsoft.sqlserver.jdbc/com/microsoft/sqlserver/jdbc/SQLServerConnection.html]
> {quote}SQLServerConnection is not thread safe, however multiple statements
> created from a single connection can be processing simultaneously in
> concurrent threads.
> {quote}
> Another interesting point to note, also to do justice to previous committers
> who have put this feature in place, is that this was not always the case. In
> the following links, you may see the same links to the older versions of the
> same pages. In the past, all vendors indicated that Connection is fully
> thread-safe; [Postgres|https://www.postgresql.org/docs/7.1/jdbc-thread.html],
> [Oracle|https://docs.oracle.com/cd/A97335_02/apps.102/a83724/tips1.htm],
> [MSSQL
> Server|https://www.javadoc.io/doc/com.microsoft.sqlserver/mssql-jdbc/6.1.0.jre7/com/microsoft/sqlserver/jdbc/SQLServerConnection.html].
>
> Hence, it was once safe to expose Connection objects via JMX given the
> thread-safety guarantees for the underlying vendor connection were in place.
> But as Vendors dropped the thread-safety guarantee one by one, it is not safe
> anymore, and may actually cause convoluted errors that pop up intermittently
> due to thread races in the JDBC driver code (see an example in the comments
> section below). Accordingly, exposing Connections via JMX shall be retired
> along with dropped support from most vendors.
> Note: the Datasource MBeans, which provide a vital set of metrics have no
> such problems as they don't depend on the underlying JDBC provider.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (DBCP-585) Connection level JMX queries result in concurrent access to connection objects, causing errors
[
https://issues.apache.org/jira/browse/DBCP-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17520227#comment-17520227
]
Phil Steitz commented on DBCP-585:
--
Nice analysis. The patch looks reasonable as a way to turn off the direct
connection access. Trying to allow this safely is an interesting problem. The
only ways that I can come up with risk performance impacts due to adding sync.
The simplest way to do it would be to force clients to obtain a lock in
DelegatingConnection.checkOpen (called by everything that actually uses a
connection). The lock would not be contended much, but it would cost something.
> Connection level JMX queries result in concurrent access to connection
> objects, causing errors
> --
>
> Key: DBCP-585
> URL: https://issues.apache.org/jira/browse/DBCP-585
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Kurtcebe Eroglu
>Priority: Major
> Attachments: 0001-DBCP-585-idea-clarification-1.patch,
> conn_instance_attrs.png, connections_attrs.png, ds_attrs.png, final.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As we expose Connection objects over JMX, they may be accessed by multiple
> threads concurrently;
> a) an application thread that borrows the Connection and uses it business as
> usual,
> b) another thread simultaneously performing a JMX query, which in turn calls
> getters on the same connection object via the MBean interface.
> Also, calls to Connection object getters are mostly delegated to the
> underlying vendor-specific connection provided by the JDBC driver. For
> example, when we make the JMX query to get the "schema" attribute of the JMX
> connection object, this is translated into a
> "java.sql.Connection.getSchema()", and passed to the vendor-specific
> Connection object by DBCP. In the case of Postgres, for example, this is
> further translated to a query "select current_schema()" and sent to the
> server.
> Hence, querying connections over JMX result in concurrent access by multiple
> threads to the underlying Connection provided by the vendors, to the point
> that these two threads may be running queries simultaneously on the same
> connection.
> However, this is not supported by any of the major database vendors. Vendor
> links on Connection objects not being threadsafe:
> - [Postgres|https://jdbc.postgresql.org/documentation/head/thread.html]
> {quote}The PostgreSQL™ JDBC driver is not thread safe. The PostgreSQL server
> is not threaded. Each connection creates a new process on the server; as such
> any concurrent requests to the process would have to be serialized. The
> driver makes no guarantees that methods on connections are synchronized. It
> will be up to the caller to synchronize calls to the driver.
> {quote}
> -
> [Oracle|https://docs.oracle.com/en/database/oracle/oracle-database/19/jjdbc/JDBC-coding-tips.html#GUID-EE479007-D105-4F82-8D51-000CBBD4BC77]
>
> {quote}Oracle strongly discourages sharing a database connection among
> multiple threads. Avoid allowing multiple threads to access a connection
> simultaneously.
> {quote}
> - [Microsoft SQL
> Server|https://www.javadoc.io/doc/com.microsoft.sqlserver/mssql-jdbc/latest/com.microsoft.sqlserver.jdbc/com/microsoft/sqlserver/jdbc/SQLServerConnection.html]
> {quote}SQLServerConnection is not thread safe, however multiple statements
> created from a single connection can be processing simultaneously in
> concurrent threads.
> {quote}
> Another interesting point to note, also to do justice to previous committers
> who have put this feature in place, is that this was not always the case. In
> the following links, you may see the same links to the older versions of the
> same pages. In the past, all vendors indicated that Connection is fully
> thread-safe; [Postgres|https://www.postgresql.org/docs/7.1/jdbc-thread.html],
> [Oracle|https://docs.oracle.com/cd/A97335_02/apps.102/a83724/tips1.htm],
> [MSSQL
> Server|https://www.javadoc.io/doc/com.microsoft.sqlserver/mssql-jdbc/6.1.0.jre7/com/microsoft/sqlserver/jdbc/SQLServerConnection.html].
>
> Hence, it was once safe to expose Connection objects via JMX given the
> thread-safety guarantees for the underlying vendor connection were in place.
> But as Vendors dropped the thread-safety guarantee one by one, it is not safe
> anymore, and may actually cause convoluted errors that pop up intermittently
> due to thread races in the JDBC driver code (see an example in the comments
> section below). Accordingly, exposing Connections via JMX shall be retired
> along with dropped support from most vendors.
> Note: the Datasource MBeans, which provide a vital set of metrics have no
> such problems as they don't depend
[jira] [Closed] (POOL-404) No way to close evictor thread
[
https://issues.apache.org/jira/browse/POOL-404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed POOL-404.
Resolution: Not A Problem
We can open an enhancement request to add global pool management, but I do not
see a pool bug here.
> No way to close evictor thread
> --
>
> Key: POOL-404
> URL: https://issues.apache.org/jira/browse/POOL-404
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.11.1
>Reporter: Patrick Barry
>Priority: Major
>
> Using GenericObjectPool> to help with
> lettuce client/redis connection management. I have everything shutting down
> cleanly except the commons-pool-evictor. I see this problem has been
> reported many times in the past, but all the changes are still not allowing
> this thread to shut down cleanly on close. I am using version 2.11.1. I
> have tried to code around this issue, but because EvictionTimer.java is so
> locked down, there is very little that can done to change the behavior of how
> this class interacts with GenericObjectPool.
> For this thread to shutdown, the taskMap has to be empty, which it never is
> in my case. So even though we call close() on the pool, this class fails to
> shutdown the embedded executor because it thinks it has more tasks.
> Looking at this code, it did remove 1 entry from taskMap, but we had many
> more in that map. Is there a way to clear this map, so it will allow this
> thread/executor to shutdown?
> {code:java}
> static synchronized void cancel(final BaseGenericObjectPool.Evictor
> evictor, final Duration timeout,
> final boolean restarting) {
> if (evictor != null) {
> evictor.cancel(); //why does this not interrupt!?
> remove(evictor);
> }
> if (!restarting && executor != null && taskMap.isEmpty()) { //<-- How do
> you force taskMap to be empty!?
> executor.shutdown();
> try {
> executor.awaitTermination(timeout.toMillis(),
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException e) {
> // Swallow
> // Significant API changes would be required to propagate this
> }
> executor.setCorePoolSize(0);
> executor = null;
> }
> }{code}
> }
> I had all these entries in the taskMap when trying to shut down:
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@73d4066e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@3c69362a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2412a42b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@45404d5
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@29138d3a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5cbe2654
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6dbcf214
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@496a31da
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@7c251f90
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@51841ac6
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5ba26eb0
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@435e60ff
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@17d32e9b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@66f0548d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2e6f610d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@1e86a5a7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@10afe71a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@741f8dbe
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@212dfd39
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@a2ddf26
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@49ede9c7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@65d57e4e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6daf7d37
> *Method calls:*
> pool.close() ->
> stopEvictor(); ->
> startEvictor(Duration.ofMillis(-1L)); ->
> EvictionTimer.cancel(evictor, evictorShutdownTimeoutDuration, false); ->
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (POOL-404) No way to close evictor thread
[
https://issues.apache.org/jira/browse/POOL-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17481215#comment-17481215
]
Phil Steitz commented on POOL-404:
--
[~ggregory] Can you see a scenario where multiple evictors can be created for
a single pool or where either of these are accessed without holding the guard
(evictionLock)? I can't make this happen and it looks to me like the
protection is good.
> No way to close evictor thread
> --
>
> Key: POOL-404
> URL: https://issues.apache.org/jira/browse/POOL-404
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.11.1
>Reporter: Patrick Barry
>Priority: Major
>
> Using GenericObjectPool> to help with
> lettuce client/redis connection management. I have everything shutting down
> cleanly except the commons-pool-evictor. I see this problem has been
> reported many times in the past, but all the changes are still not allowing
> this thread to shut down cleanly on close. I am using version 2.11.1. I
> have tried to code around this issue, but because EvictionTimer.java is so
> locked down, there is very little that can done to change the behavior of how
> this class interacts with GenericObjectPool.
> For this thread to shutdown, the taskMap has to be empty, which it never is
> in my case. So even though we call close() on the pool, this class fails to
> shutdown the embedded executor because it thinks it has more tasks.
> Looking at this code, it did remove 1 entry from taskMap, but we had many
> more in that map. Is there a way to clear this map, so it will allow this
> thread/executor to shutdown?
> {code:java}
> static synchronized void cancel(final BaseGenericObjectPool.Evictor
> evictor, final Duration timeout,
> final boolean restarting) {
> if (evictor != null) {
> evictor.cancel(); //why does this not interrupt!?
> remove(evictor);
> }
> if (!restarting && executor != null && taskMap.isEmpty()) { //<-- How do
> you force taskMap to be empty!?
> executor.shutdown();
> try {
> executor.awaitTermination(timeout.toMillis(),
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException e) {
> // Swallow
> // Significant API changes would be required to propagate this
> }
> executor.setCorePoolSize(0);
> executor = null;
> }
> }{code}
> }
> I had all these entries in the taskMap when trying to shut down:
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@73d4066e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@3c69362a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2412a42b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@45404d5
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@29138d3a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5cbe2654
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6dbcf214
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@496a31da
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@7c251f90
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@51841ac6
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5ba26eb0
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@435e60ff
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@17d32e9b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@66f0548d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2e6f610d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@1e86a5a7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@10afe71a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@741f8dbe
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@212dfd39
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@a2ddf26
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@49ede9c7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@65d57e4e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6daf7d37
> *Method calls:*
> pool.close() ->
> stopEvictor(); ->
> startEvictor(Duration.ofMillis(-1L)); ->
> EvictionTimer.cancel(evictor, evictorShutdownTimeoutDuration, false); ->
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (POOL-404) No way to close evictor thread
[
https://issues.apache.org/jira/browse/POOL-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480744#comment-17480744
]
Phil Steitz commented on POOL-404:
--
[~patrickjamesbarry] That is interesting. Could be we actually have a pool bug
here. What is the config for the singleton pool? Do you change the evictor
settings after initialization? Other than not seeing more than one via
jconsole, are you really sure there are not multiple pools being created. When
you say the constructor is only called once, how do you know that? Not being
argumentative here, I just want to make sure I understand the context. What I
don't understand is that a single pool should not be able to have multiple
evictors running concurrently and if you have only one and the EvictionTimer's
taskMap has multiple tasks in it, something is wrong. Can you possibly create
a unit test that makes this happen? I will try to do this myself too. I will
also look at the lettuce code. What is the unit test creating the output above?
> No way to close evictor thread
> --
>
> Key: POOL-404
> URL: https://issues.apache.org/jira/browse/POOL-404
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.11.1
>Reporter: Patrick Barry
>Priority: Major
>
> Using GenericObjectPool> to help with
> lettuce client/redis connection management. I have everything shutting down
> cleanly except the commons-pool-evictor. I see this problem has been
> reported many times in the past, but all the changes are still not allowing
> this thread to shut down cleanly on close. I am using version 2.11.1. I
> have tried to code around this issue, but because EvictionTimer.java is so
> locked down, there is very little that can done to change the behavior of how
> this class interacts with GenericObjectPool.
> For this thread to shutdown, the taskMap has to be empty, which it never is
> in my case. So even though we call close() on the pool, this class fails to
> shutdown the embedded executor because it thinks it has more tasks.
> Looking at this code, it did remove 1 entry from taskMap, but we had many
> more in that map. Is there a way to clear this map, so it will allow this
> thread/executor to shutdown?
> {code:java}
> static synchronized void cancel(final BaseGenericObjectPool.Evictor
> evictor, final Duration timeout,
> final boolean restarting) {
> if (evictor != null) {
> evictor.cancel(); //why does this not interrupt!?
> remove(evictor);
> }
> if (!restarting && executor != null && taskMap.isEmpty()) { //<-- How do
> you force taskMap to be empty!?
> executor.shutdown();
> try {
> executor.awaitTermination(timeout.toMillis(),
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException e) {
> // Swallow
> // Significant API changes would be required to propagate this
> }
> executor.setCorePoolSize(0);
> executor = null;
> }
> }{code}
> }
> I had all these entries in the taskMap when trying to shut down:
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@73d4066e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@3c69362a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2412a42b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@45404d5
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@29138d3a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5cbe2654
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6dbcf214
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@496a31da
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@7c251f90
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@51841ac6
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5ba26eb0
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@435e60ff
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@17d32e9b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@66f0548d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2e6f610d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@1e86a5a7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@10afe71a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@741f8dbe
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@212dfd39
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@a2ddf26
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@49ede9c7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@65d57e4e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6daf7d37
> *Method calls:*
> pool.close() ->
> stopEvictor(); ->
> startEvictor(Duration.ofMillis(-1L)); ->
>
[jira] [Commented] (POOL-404) No way to close evictor thread
[
https://issues.apache.org/jira/browse/POOL-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480706#comment-17480706
]
Phil Steitz commented on POOL-404:
--
Yes that is what I meant by the closeAll option. Still seems a little smelly
to me. Like if jdbc Connection were to expose a static closeAll. Much better
to just have the client manage the concern of closing the pools. Or if we
really want to add global pool management, add some kind of (non-static) pool
manager that clients explicitly instantiate and it maintains a registry. But
most practical cases (likely incl OPs) can be managed by GKOP.
> No way to close evictor thread
> --
>
> Key: POOL-404
> URL: https://issues.apache.org/jira/browse/POOL-404
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.11.1
>Reporter: Patrick Barry
>Priority: Major
>
> Using GenericObjectPool> to help with
> lettuce client/redis connection management. I have everything shutting down
> cleanly except the commons-pool-evictor. I see this problem has been
> reported many times in the past, but all the changes are still not allowing
> this thread to shut down cleanly on close. I am using version 2.11.1. I
> have tried to code around this issue, but because EvictionTimer.java is so
> locked down, there is very little that can done to change the behavior of how
> this class interacts with GenericObjectPool.
> For this thread to shutdown, the taskMap has to be empty, which it never is
> in my case. So even though we call close() on the pool, this class fails to
> shutdown the embedded executor because it thinks it has more tasks.
> Looking at this code, it did remove 1 entry from taskMap, but we had many
> more in that map. Is there a way to clear this map, so it will allow this
> thread/executor to shutdown?
> {code:java}
> static synchronized void cancel(final BaseGenericObjectPool.Evictor
> evictor, final Duration timeout,
> final boolean restarting) {
> if (evictor != null) {
> evictor.cancel(); //why does this not interrupt!?
> remove(evictor);
> }
> if (!restarting && executor != null && taskMap.isEmpty()) { //<-- How do
> you force taskMap to be empty!?
> executor.shutdown();
> try {
> executor.awaitTermination(timeout.toMillis(),
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException e) {
> // Swallow
> // Significant API changes would be required to propagate this
> }
> executor.setCorePoolSize(0);
> executor = null;
> }
> }{code}
> }
> I had all these entries in the taskMap when trying to shut down:
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@73d4066e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@3c69362a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2412a42b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@45404d5
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@29138d3a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5cbe2654
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6dbcf214
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@496a31da
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@7c251f90
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@51841ac6
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5ba26eb0
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@435e60ff
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@17d32e9b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@66f0548d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2e6f610d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@1e86a5a7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@10afe71a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@741f8dbe
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@212dfd39
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@a2ddf26
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@49ede9c7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@65d57e4e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6daf7d37
> *Method calls:*
> pool.close() ->
> stopEvictor(); ->
> startEvictor(Duration.ofMillis(-1L)); ->
> EvictionTimer.cancel(evictor, evictorShutdownTimeoutDuration, false); ->
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (POOL-404) No way to close evictor thread
[
https://issues.apache.org/jira/browse/POOL-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480250#comment-17480250
]
Phil Steitz edited comment on POOL-404 at 1/22/22, 5:26 PM:
That would have to be a *global* JVM-level close that would not be advisable
[~ggregory] . [~patrickjamesbarry] , those references are to evictors for
pools that appear to be open. Are you sure that your code is calling close on
each pool that it creates?
The EvictionTimer manages pool-specific evictor tasks at the JVM level. The
task map that it maintains holds references to evictors associated with open
pools. The cancel method removes the evictor that it is passed. If the map is
not empty, that means that there are other pools open with evictors running.
If all pools are closed, all evictors will be cancelled and the deamon thread
will terminate. For this to work, all pools have to get closed. There are two
things we could do here to make it easier for clients: 0) enable the
EvictionTimer to hold references to the pools and expose an iterator over all
open pools so client shutdown code could get a list of them and cleanly shut
them down 1) expose a closeAllPools method that closes all open pools. Or 1)
could just kill running evictors, leaving the pools open but no longer
consistent with their configs. This is a little smelly as what both really
amount to is using EvictionTimer to allow clients to manage pools globally. It
might be better to introduce a separate class for that or to just recommend
that when people want to manage a collection of pools, use
GenericKeyedObjectPool instead.
was (Author: psteitz):
That would have to be a *global* JVM-level close that would not be advisable
[~ggregory] . [~patrickjamesbarry] , those references are to evictors for
pools that appear to be open. Are you sure that your code is calling close on
each pool that it creates?
> No way to close evictor thread
> --
>
> Key: POOL-404
> URL: https://issues.apache.org/jira/browse/POOL-404
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.11.1
>Reporter: Patrick Barry
>Priority: Major
>
> Using GenericObjectPool> to help with
> lettuce client/redis connection management. I have everything shutting down
> cleanly except the commons-pool-evictor. I see this problem has been
> reported many times in the past, but all the changes are still not allowing
> this thread to shut down cleanly on close. I am using version 2.11.1. I
> have tried to code around this issue, but because EvictionTimer.java is so
> locked down, there is very little that can done to change the behavior of how
> this class interacts with GenericObjectPool.
> For this thread to shutdown, the taskMap has to be empty, which it never is
> in my case. So even though we call close() on the pool, this class fails to
> shutdown the embedded executor because it thinks it has more tasks.
> Looking at this code, it did remove 1 entry from taskMap, but we had many
> more in that map. Is there a way to clear this map, so it will allow this
> thread/executor to shutdown?
> {code:java}
> static synchronized void cancel(final BaseGenericObjectPool.Evictor
> evictor, final Duration timeout,
> final boolean restarting) {
> if (evictor != null) {
> evictor.cancel(); //why does this not interrupt!?
> remove(evictor);
> }
> if (!restarting && executor != null && taskMap.isEmpty()) { //<-- How do
> you force taskMap to be empty!?
> executor.shutdown();
> try {
> executor.awaitTermination(timeout.toMillis(),
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException e) {
> // Swallow
> // Significant API changes would be required to propagate this
> }
> executor.setCorePoolSize(0);
> executor = null;
> }
> }{code}
> }
> I had all these entries in the taskMap when trying to shut down:
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@73d4066e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@3c69362a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2412a42b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@45404d5
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@29138d3a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5cbe2654
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6dbcf214
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@496a31da
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@7c251f90
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@51841ac6
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5ba26eb0
>
[jira] [Commented] (POOL-404) No way to close evictor thread
[
https://issues.apache.org/jira/browse/POOL-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480250#comment-17480250
]
Phil Steitz commented on POOL-404:
--
That would have to be a *global* JVM-level close that would not be advisable
[~ggregory] . [~patrickjamesbarry] , those references are to evictors for
pools that appear to be open. Are you sure that your code is calling close on
each pool that it creates?
> No way to close evictor thread
> --
>
> Key: POOL-404
> URL: https://issues.apache.org/jira/browse/POOL-404
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.11.1
>Reporter: Patrick Barry
>Priority: Major
>
> Using GenericObjectPool> to help with
> lettuce client/redis connection management. I have everything shutting down
> cleanly except the commons-pool-evictor. I see this problem has been
> reported many times in the past, but all the changes are still not allowing
> this thread to shut down cleanly on close. I am using version 2.11.1. I
> have tried to code around this issue, but because EvictionTimer.java is so
> locked down, there is very little that can done to change the behavior of how
> this class interacts with GenericObjectPool.
> For this thread to shutdown, the taskMap has to be empty, which it never is
> in my case. So even though we call close() on the pool, this class fails to
> shutdown the embedded executor because it thinks it has more tasks.
> Looking at this code, it did remove 1 entry from taskMap, but we had many
> more in that map. Is there a way to clear this map, so it will allow this
> thread/executor to shutdown?
> {code:java}
> static synchronized void cancel(final BaseGenericObjectPool.Evictor
> evictor, final Duration timeout,
> final boolean restarting) {
> if (evictor != null) {
> evictor.cancel(); //why does this not interrupt!?
> remove(evictor);
> }
> if (!restarting && executor != null && taskMap.isEmpty()) { //<-- How do
> you force taskMap to be empty!?
> executor.shutdown();
> try {
> executor.awaitTermination(timeout.toMillis(),
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException e) {
> // Swallow
> // Significant API changes would be required to propagate this
> }
> executor.setCorePoolSize(0);
> executor = null;
> }
> }{code}
> }
> I had all these entries in the taskMap when trying to shut down:
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@73d4066e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@3c69362a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2412a42b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@45404d5
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@29138d3a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5cbe2654
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6dbcf214
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@496a31da
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@7c251f90
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@51841ac6
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5ba26eb0
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@435e60ff
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@17d32e9b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@66f0548d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2e6f610d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@1e86a5a7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@10afe71a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@741f8dbe
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@212dfd39
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@a2ddf26
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@49ede9c7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@65d57e4e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6daf7d37
> *Method calls:*
> pool.close() ->
> stopEvictor(); ->
> startEvictor(Duration.ofMillis(-1L)); ->
> EvictionTimer.cancel(evictor, evictorShutdownTimeoutDuration, false); ->
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (POOL-403) GenericKeyedObjectPool getNumActive and getNumIdle Javadoc do not match behaviour
[ https://issues.apache.org/jira/browse/POOL-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454139#comment-17454139 ] Phil Steitz commented on POOL-403: -- This is being picked up from the KeyedObjectPool interface javadoc. I am not sure this is actually a bug. In GKOP, this information is *always* available and it correctly returns 0 when it should. The interface spec allows alternative implementations to return -1 when the info is not available. > GenericKeyedObjectPool getNumActive and getNumIdle Javadoc do not match > behaviour > - > > Key: POOL-403 > URL: https://issues.apache.org/jira/browse/POOL-403 > Project: Commons Pool > Issue Type: Improvement >Affects Versions: 2.11.1 >Reporter: KON-SUN-TACK >Priority: Minor > > In GenericKeyedObjectPool Javadoc, getNumActive(K key) and getNumIdle(K key) > states: > "Returns a negative value if this information is not available." > But it seems they actually return 0 instead. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (POOL-393) BaseGenericObjectPool.jmxRegister may cost too much time
[
https://issues.apache.org/jira/browse/POOL-393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443194#comment-17443194
]
Phil Steitz edited comment on POOL-393 at 11/14/21, 4:55 PM:
-
I verified that with the 2.11.1 code it can take 9+ seconds to register 1000
pools due to all of the exception processing. Changing to use an AtomicLong to
keep track of the last used suffix reduces it by at least 10x. Proposed fix is
in [https://github.com/apache/commons-pool/pull/115].
I should note that the PR changes behavior in some cases. Current code
"reuses" suffixes when pools are unregistered. After the PR, new pools always
get new suffixes.
was (Author: psteitz):
I verified that with the 2.11.1 code it can take 9+ seconds to register 1000
pools due to all of the exception processing. Changing to use an AtomicLong to
keep track of the last used suffix reduces it by at least 10x. Proposed fix is
in https://github.com/apache/commons-pool/pull/115.
> BaseGenericObjectPool.jmxRegister may cost too much time
>
>
> Key: POOL-393
> URL: https://issues.apache.org/jira/browse/POOL-393
> Project: Commons Pool
> Issue Type: Improvement
>Affects Versions: 2.4.2
>Reporter: Shichao Yuan
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
>
> When creating many pools, I find that it tasks too much time to register jmx.
> In the code, the ObjectName's postfix always starts with 1, so many
> InstanceAlreadyExistsExceptions may be thrown before registered successfully.
> Maybe a random number is a better choice, or a atomic long.
> {quote}private ObjectName jmxRegister(BaseObjectPoolConfig config,
> String jmxNameBase, String jmxNamePrefix) {
> ObjectName objectName = null;
> MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
> int i = 1;
> boolean registered = false;
> String base = config.getJmxNameBase();
> if (base == null)
> Unknown macro: \{ base = jmxNameBase; }
> while (!registered) {
> try {
> ObjectName objName;
> // Skip the numeric suffix for the first pool in case there is
> // only one so the names are cleaner.
> if (i == 1)
> Unknown macro: \{ objName = new ObjectName(base + jmxNamePrefix); }
> else
> Unknown macro: \{ objName = new ObjectName(base + jmxNamePrefix + i); }
> mbs.registerMBean(this, objName);
> objectName = objName;
> registered = true;
> } catch (MalformedObjectNameException e) {
> if (BaseObjectPoolConfig.DEFAULT_JMX_NAME_PREFIX.equals(
> jmxNamePrefix) && jmxNameBase.equals(base))
> Unknown macro: \{ // Shouldn't happen. Skip registration if it does.
> registered = true; }
> else
> Unknown macro: \{ // Must be an invalid name. Use the defaults instead.
> jmxNamePrefix = BaseObjectPoolConfig.DEFAULT_JMX_NAME_PREFIX; base =
> jmxNameBase; }
> } catch (InstanceAlreadyExistsException e)
> Unknown macro: \{ // Increment the index and try again i++; }
> catch (MBeanRegistrationException e)
> Unknown macro: \{ // Shouldn't happen. Skip registration if it does.
> registered = true; }
> catch (NotCompliantMBeanException e)
> }
> return objectName;
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (POOL-393) BaseGenericObjectPool.jmxRegister may cost too much time
[
https://issues.apache.org/jira/browse/POOL-393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443194#comment-17443194
]
Phil Steitz commented on POOL-393:
--
I verified that with the 2.11.1 code it can take 9+ seconds to register 1000
pools due to all of the exception processing. Changing to use an AtomicLong to
keep track of the last used suffix reduces it by at least 10x. Proposed fix is
in https://github.com/apache/commons-pool/pull/115.
> BaseGenericObjectPool.jmxRegister may cost too much time
>
>
> Key: POOL-393
> URL: https://issues.apache.org/jira/browse/POOL-393
> Project: Commons Pool
> Issue Type: Improvement
>Affects Versions: 2.4.2
>Reporter: Shichao Yuan
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
>
> When creating many pools, I find that it tasks too much time to register jmx.
> In the code, the ObjectName's postfix always starts with 1, so many
> InstanceAlreadyExistsExceptions may be thrown before registered successfully.
> Maybe a random number is a better choice, or a atomic long.
> {quote}private ObjectName jmxRegister(BaseObjectPoolConfig config,
> String jmxNameBase, String jmxNamePrefix) {
> ObjectName objectName = null;
> MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
> int i = 1;
> boolean registered = false;
> String base = config.getJmxNameBase();
> if (base == null)
> Unknown macro: \{ base = jmxNameBase; }
> while (!registered) {
> try {
> ObjectName objName;
> // Skip the numeric suffix for the first pool in case there is
> // only one so the names are cleaner.
> if (i == 1)
> Unknown macro: \{ objName = new ObjectName(base + jmxNamePrefix); }
> else
> Unknown macro: \{ objName = new ObjectName(base + jmxNamePrefix + i); }
> mbs.registerMBean(this, objName);
> objectName = objName;
> registered = true;
> } catch (MalformedObjectNameException e) {
> if (BaseObjectPoolConfig.DEFAULT_JMX_NAME_PREFIX.equals(
> jmxNamePrefix) && jmxNameBase.equals(base))
> Unknown macro: \{ // Shouldn't happen. Skip registration if it does.
> registered = true; }
> else
> Unknown macro: \{ // Must be an invalid name. Use the defaults instead.
> jmxNamePrefix = BaseObjectPoolConfig.DEFAULT_JMX_NAME_PREFIX; base =
> jmxNameBase; }
> } catch (InstanceAlreadyExistsException e)
> Unknown macro: \{ // Increment the index and try again i++; }
> catch (MBeanRegistrationException e)
> Unknown macro: \{ // Shouldn't happen. Skip registration if it does.
> registered = true; }
> catch (NotCompliantMBeanException e)
> }
> return objectName;
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (POOL-394) GenericKeyedObjectPool doesn't have getKeys() implemented
[ https://issues.apache.org/jira/browse/POOL-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443182#comment-17443182 ] Phil Steitz commented on POOL-394: -- I have not run into this requirement in my own apps, but I can see how it would be convenient for users. It is in fact already indirectly available via ``listAllObjects.getKeys()``. I see no reason not to expose it directly with the same semantics / disclaimer - you get a copy that does not necessarily represent a valid snapshot at any point in time. > GenericKeyedObjectPool doesn't have getKeys() implemented > - > > Key: POOL-394 > URL: https://issues.apache.org/jira/browse/POOL-394 > Project: Commons Pool > Issue Type: Bug >Affects Versions: 2.4.2 >Reporter: Vamsi Pavan Kumar Sanka >Priority: Major > > getKeys() not really implemented as specified here: > [https://commons.apache.org/proper/commons-pool2/apidocs/org/apache/commons/pool2/impl/GenericKeyedObjectPool.html#getKeys(]) > This API is needed in various scenarios. Current Jira is to expose this API. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (POOL-401) GKOP invalidateObject should make capacity available to all keyed pools
Phil Steitz created POOL-401: Summary: GKOP invalidateObject should make capacity available to all keyed pools Key: POOL-401 URL: https://issues.apache.org/jira/browse/POOL-401 Project: Commons Pool Issue Type: Bug Reporter: Phil Steitz GKOP invalidateObject currently only creates and adds objects if there are borrowers waiting for the keyed pool that the instance being invalidated came from. If there are no take waiters on that pool, or there is a more heavily loaded pool, the (total) capacity should be reused in a different pool. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (POOL-391) GenericKeyedObjectPool is not thread safe when invoke method `borrowObject` and `destroy` simultaneously
[
https://issues.apache.org/jira/browse/POOL-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442354#comment-17442354
]
Phil Steitz commented on POOL-391:
--
A fix for this is in [https://github.com/apache/commons-pool/pull/113]
The lack of "atomicity" is by design, to avoid synchronization delays waiting
for creates to complete. Other public methods that create capacity (return,
invalidate, evict, etc) call reuseCapacity, which allocates freed capacity to
pools that have borrowers waiting. The PR modifies clear() to (optionally) do
this.
> GenericKeyedObjectPool is not thread safe when invoke method `borrowObject`
> and `destroy` simultaneously
> -
>
> Key: POOL-391
> URL: https://issues.apache.org/jira/browse/POOL-391
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.4.2, 2.5.0, 2.6.0, 2.7.0, 2.8.0, 2.9.0
>Reporter: Codievilky August
>Priority: Blocker
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The method `brrowObject` is not thread safe with `destroy` or `clear` method.
> The reason is when use GenericKeyedObjectPool#destroy method，it did not
> ensure the *Atomicity* of destroy object from the ObjectDeque。
> This may cause in the GenericKeyedObjectPool#borrowObject method，may get the
> wrong number of GenericKeyedObjectPool.ObjectDeque#getCreateCount when need
> to create. And the GenericKeyedObjectPool#borrowObject will block until
> timeout.
> Here is the sample test code to recur the bug:
> {code:java}
> // code placeholder
> public class CommonPoolMultiThreadTest {
> public static void main(String[] args) {
> GenericKeyedObjectPoolConfig config = new GenericKeyedObjectPoolConfig();
> config.setMaxTotalPerKey(1);
> config.setMinIdlePerKey(0);
> config.setMaxIdlePerKey(-1);
> config.setMaxTotal(-1);
> config.setMaxWaitMillis(TimeUnit.SECONDS.toMillis(5));
> GenericKeyedObjectPool testPool = new
> GenericKeyedObjectPool<>(
> new KeyedPooledObjectFactory() {
> @Override
> public PooledObject makeObject(Integer key) throws
> Exception {
> System.out.println("start to create");
> return new DefaultPooledObject<>(10);
> } @Override
> public void destroyObject(Integer key, PooledObject p)
> throws Exception {
> System.out.println("start to destroy");
> Thread.sleep(2000);
> } @Override
> public boolean validateObject(Integer key, PooledObject p)
> {
> return true;
> } @Override
> public void activateObject(Integer key, PooledObject p)
> throws Exception {
> // do nothing
> } @Override
> public void passivateObject(Integer key, PooledObject p)
> throws Exception {
> // do nothing
> }
> }, config
> );
> int borrowKey = 10;
> Thread t = new Thread(() -> {
> try {
> while (true) {
> Integer integer = testPool.borrowObject(borrowKey);
> testPool.returnObject(borrowKey, integer);
> Thread.sleep(10);
> }
> } catch (Exception e) {
> e.printStackTrace();
> System.exit(1);
> }
> });
> Thread t2 = new Thread(() -> {
> try {
> while (true) {
> testPool.clear(borrowKey);
> Thread.sleep(10);
> }
> } catch (Exception e) {
> e.printStackTrace();
> System.exit(1);
> }
> });
> t.start();
> t2.start();
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (POOL-396) Exceptions in validation can cause an object's lifecycle to be stuck in EVICTION state
[
https://issues.apache.org/jira/browse/POOL-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17366264#comment-17366264
]
Phil Steitz commented on POOL-396:
--
Sorry for the slow response. Really nice work on this [~jeremyk-91]. One
small change (responding to your question above) is that I would just rethrow
the original exception rather than wrapping in a NSEE. That makes sense on
borrow, but here it's probably better to just propagate whatever was thrown by
the factory.
> Exceptions in validation can cause an object's lifecycle to be stuck in
> EVICTION state
> --
>
> Key: POOL-396
> URL: https://issues.apache.org/jira/browse/POOL-396
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.4.2, 2.9.0
>Reporter: Jeremy Kong
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, if validation of an object throws an exception when testing an
> idle object during eviction, an exception will be thrown from the evict()
> method. This causes the object to be left in the EVICTION state, but still be
> present in the idleObjects queue.
> Future runs of the eviction thread will run into an infinite loop, because
> they pick up the afflicted object from the queue, fail to set its state to
> EVICTION, don't count it as a test, and repeat ad infinitum. This infinite
> loop means that no further evictions will take place; it also causes
> increases in garbage collection owing to allocation of the EvictionIterator
> on each loop iteration.
> This test added to TestGenericObjectPool (plus a small change allowing
> exceptions to be thrown from the validateObject method of SimpleFactory)
> reproduces the issue.
> {code:java}
> @Test
> @Timeout(value = 6, unit = TimeUnit.MILLISECONDS)
> public void testExceptionInValidationDuringEviction() throws Exception {
> genericObjectPool.setMaxIdle(1);
> genericObjectPool.setMaxTotal(2);
> genericObjectPool.setNumTestsPerEvictionRun(1);
> genericObjectPool.setMinEvictableIdleTime(Duration.ofMillis(0));
> genericObjectPool.setTestWhileIdle(true);
> String active = genericObjectPool.borrowObject();
> genericObjectPool.returnObject(active);
> simpleFactory.setThrowExceptionOnValidate(true);
> try {
> genericObjectPool.evict();
> } catch (Exception e) {
> // OK that a given run fails. But the object should still be evicted.
> }
> genericObjectPool.evict(); // currently fails: causes an infinite loop!
> assertEquals(0, genericObjectPool.getNumActive());
> assertEquals(0, genericObjectPool.getNumIdle());
> }
> {code}
> It seems that protection against exceptions thrown by validation code exist
> elsewhere (e.g. in borrowObject()); similarly, protections against exceptions
> thrown by user eviction policy code exist in evict(). So it looks like the
> testWhileIdle() portion of evict() needs to similarly be protected.
> Thanks!
> Jeremy
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (POOL-396) Exceptions in validation can cause an object's lifecycle to be stuck in EVICTION state
[
https://issues.apache.org/jira/browse/POOL-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364356#comment-17364356
]
Phil Steitz commented on POOL-396:
--
After sleeping on this, I think it is probably better to go ahead and destroy
the instance before rethrowing the exception. We should also not decrement the
numtests counter in this case.
We should probably add prominent docs somewhere (site and javadoc) pointing out
that factory methods should handle their own exceptions. We try to do
reasonable things when they throw, but this case is a good example where we
need the factory to tell us what to do.
> Exceptions in validation can cause an object's lifecycle to be stuck in
> EVICTION state
> --
>
> Key: POOL-396
> URL: https://issues.apache.org/jira/browse/POOL-396
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.4.2, 2.9.0
>Reporter: Jeremy Kong
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, if validation of an object throws an exception when testing an
> idle object during eviction, an exception will be thrown from the evict()
> method. This causes the object to be left in the EVICTION state, but still be
> present in the idleObjects queue.
> Future runs of the eviction thread will run into an infinite loop, because
> they pick up the afflicted object from the queue, fail to set its state to
> EVICTION, don't count it as a test, and repeat ad infinitum. This infinite
> loop means that no further evictions will take place; it also causes
> increases in garbage collection owing to allocation of the EvictionIterator
> on each loop iteration.
> This test added to TestGenericObjectPool (plus a small change allowing
> exceptions to be thrown from the validateObject method of SimpleFactory)
> reproduces the issue.
> {code:java}
> @Test
> @Timeout(value = 6, unit = TimeUnit.MILLISECONDS)
> public void testExceptionInValidationDuringEviction() throws Exception {
> genericObjectPool.setMaxIdle(1);
> genericObjectPool.setMaxTotal(2);
> genericObjectPool.setNumTestsPerEvictionRun(1);
> genericObjectPool.setMinEvictableIdleTime(Duration.ofMillis(0));
> genericObjectPool.setTestWhileIdle(true);
> String active = genericObjectPool.borrowObject();
> genericObjectPool.returnObject(active);
> simpleFactory.setThrowExceptionOnValidate(true);
> try {
> genericObjectPool.evict();
> } catch (Exception e) {
> // OK that a given run fails. But the object should still be evicted.
> }
> genericObjectPool.evict(); // currently fails: causes an infinite loop!
> assertEquals(0, genericObjectPool.getNumActive());
> assertEquals(0, genericObjectPool.getNumIdle());
> }
> {code}
> It seems that protection against exceptions thrown by validation code exist
> elsewhere (e.g. in borrowObject()); similarly, protections against exceptions
> thrown by user eviction policy code exist in evict(). So it looks like the
> testWhileIdle() portion of evict() needs to similarly be protected.
> Thanks!
> Jeremy
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (POOL-396) Exceptions in validation can cause an object's lifecycle to be stuck in EVICTION state
[
https://issues.apache.org/jira/browse/POOL-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364015#comment-17364015
]
Phil Steitz commented on POOL-396:
--
Nice work on the report and patch. Thanks especially for the unit test
showing the bug. I have a couple of comments on the patch:
# We should remember to apply an analogous patch to GenericKeyedObjectPool,
which is subject to the same issue.
# I think it would be better to fix the eviction state of the object under
test and rethrow the exception (like what borrowObject does when the instance
failing validation has just been created) rather than swallowing the exception
and destroying the instance under test.
When I say "fix the eviction state" I mean basically just call endEvictionTest.
My reasoning for this is that validation exceptions often indicate
infrastructure issues that may be transient and that clients may want to know
about.
> Exceptions in validation can cause an object's lifecycle to be stuck in
> EVICTION state
> --
>
> Key: POOL-396
> URL: https://issues.apache.org/jira/browse/POOL-396
> Project: Commons Pool
> Issue Type: Bug
>Affects Versions: 2.4.2, 2.9.0
>Reporter: Jeremy Kong
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, if validation of an object throws an exception when testing an
> idle object during eviction, an exception will be thrown from the evict()
> method. This causes the object to be left in the EVICTION state, but still be
> present in the idleObjects queue.
> Future runs of the eviction thread will run into an infinite loop, because
> they pick up the afflicted object from the queue, fail to set its state to
> EVICTION, don't count it as a test, and repeat ad infinitum. This infinite
> loop means that no further evictions will take place; it also causes
> increases in garbage collection owing to allocation of the EvictionIterator
> on each loop iteration.
> This test added to TestGenericObjectPool (plus a small change allowing
> exceptions to be thrown from the validateObject method of SimpleFactory)
> reproduces the issue.
> {code:java}
> @Test
> @Timeout(value = 6, unit = TimeUnit.MILLISECONDS)
> public void testExceptionInValidationDuringEviction() throws Exception {
> genericObjectPool.setMaxIdle(1);
> genericObjectPool.setMaxTotal(2);
> genericObjectPool.setNumTestsPerEvictionRun(1);
> genericObjectPool.setMinEvictableIdleTime(Duration.ofMillis(0));
> genericObjectPool.setTestWhileIdle(true);
> String active = genericObjectPool.borrowObject();
> genericObjectPool.returnObject(active);
> simpleFactory.setThrowExceptionOnValidate(true);
> try {
> genericObjectPool.evict();
> } catch (Exception e) {
> // OK that a given run fails. But the object should still be evicted.
> }
> genericObjectPool.evict(); // currently fails: causes an infinite loop!
> assertEquals(0, genericObjectPool.getNumActive());
> assertEquals(0, genericObjectPool.getNumIdle());
> }
> {code}
> It seems that protection against exceptions thrown by validation code exist
> elsewhere (e.g. in borrowObject()); similarly, protections against exceptions
> thrown by user eviction policy code exist in evict(). So it looks like the
> testWhileIdle() portion of evict() needs to similarly be protected.
> Thanks!
> Jeremy
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (DBCP-572) timed out connections remain active in the pool
[
https://issues.apache.org/jira/browse/DBCP-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz updated DBCP-572:
-
Assignee: (was: Phil Steitz)
> timed out connections remain active in the pool
> ---
>
> Key: DBCP-572
> URL: https://issues.apache.org/jira/browse/DBCP-572
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Erol Guven
>Priority: Major
>
> when the database does not respond within a time-out period, the connection
> pool seems to still keep the connection in its pool and consider it active.
> These active connections are never closed and seems to be kept in the pool
> forever.
> To reproduce:
> * create a BasicDataSource
> * suspend the database process using {{kill -STOP}} signal
> * get a connection from multiple threads simultaneously which will timeout
> * inspect org.apache.commons.pool2.impl.GenericObjectPool.getNumActive()
> The active count is equivalent to the number of timed out connections.
> The active count never goes down.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DBCP-572) timed out connections remain active in the pool
[
https://issues.apache.org/jira/browse/DBCP-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291763#comment-17291763
]
Phil Steitz commented on DBCP-572:
--
This is not a bug. By default, the pool does not try to keep connections alive
while they are idle. If you want to do that, configure one or both of
connection validation or testWhileIdle. This is really a question for the user
list.
> timed out connections remain active in the pool
> ---
>
> Key: DBCP-572
> URL: https://issues.apache.org/jira/browse/DBCP-572
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Erol Guven
>Assignee: Phil Steitz
>Priority: Major
>
> when the database does not respond within a time-out period, the connection
> pool seems to still keep the connection in its pool and consider it active.
> These active connections are never closed and seems to be kept in the pool
> forever.
> To reproduce:
> * create a BasicDataSource
> * suspend the database process using {{kill -STOP}} signal
> * get a connection from multiple threads simultaneously which will timeout
> * inspect org.apache.commons.pool2.impl.GenericObjectPool.getNumActive()
> The active count is equivalent to the number of timed out connections.
> The active count never goes down.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (DBCP-572) timed out connections remain active in the pool
[
https://issues.apache.org/jira/browse/DBCP-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz updated DBCP-572:
-
Assignee: Phil Steitz
> timed out connections remain active in the pool
> ---
>
> Key: DBCP-572
> URL: https://issues.apache.org/jira/browse/DBCP-572
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Erol Guven
>Assignee: Phil Steitz
>Priority: Major
>
> when the database does not respond within a time-out period, the connection
> pool seems to still keep the connection in its pool and consider it active.
> These active connections are never closed and seems to be kept in the pool
> forever.
> To reproduce:
> * create a BasicDataSource
> * suspend the database process using {{kill -STOP}} signal
> * get a connection from multiple threads simultaneously which will timeout
> * inspect org.apache.commons.pool2.impl.GenericObjectPool.getNumActive()
> The active count is equivalent to the number of timed out connections.
> The active count never goes down.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (POOL-390) Add a max age to pool entries
[ https://issues.apache.org/jira/browse/POOL-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246147#comment-17246147 ] Phil Steitz commented on POOL-390: -- Looks like a useful feature to me . DBCP does this in its object factory, PoolableConnectionFactory. Assuming you are extending BasePooledObjectFactory and using DefaultPooledObjects in the pool, the creation time is tracked and you can just check it for max age in the factory's activation method. See o.a.c.dbcp2.PoolableConnectionFactory#activateObject for a way to do this with pool2 as is. The pool enhancement that might make sense is to either add a maxLifetime property to BasePooledObjectFactory and have the default activateObject impl do what DBCP's pcf does or just make maxLifetime a GOP/GKOP property and have borrow and return operations do the checks (DBCP2 BasePooledObject exposes createTime). The second is probably easier for users and also allows a little better control. > Add a max age to pool entries > - > > Key: POOL-390 > URL: https://issues.apache.org/jira/browse/POOL-390 > Project: Commons Pool > Issue Type: Improvement >Reporter: reggie >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > It would be nice if pool2 allowed configuration of a "max-age" parameter on a > pool entry. When this is set, at checkout time, if the max age is exceeded, > this entry would be discarded. It would be nice to do this instead of an > evictor thread in cases where pool access may be very sporadic. The evictor > thread model could run many times and do nothing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DBCP-567) Use abort rather than close to clean up abandoned connections
Phil Steitz created DBCP-567: Summary: Use abort rather than close to clean up abandoned connections Key: DBCP-567 URL: https://issues.apache.org/jira/browse/DBCP-567 Project: Commons DBCP Issue Type: Improvement Reporter: Phil Steitz Using abort rather than close to disconnect abandoned connections will make it less likely that pool threads get stuck waiting for locks on abandoned connections. To implement this, we will need the changes in [Pool 387|https://issues.apache.org/jira/browse/POOL-387] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (POOL-387) Object factory destroy method should carry information on activation context
Phil Steitz created POOL-387: Summary: Object factory destroy method should carry information on activation context Key: POOL-387 URL: https://issues.apache.org/jira/browse/POOL-387 Project: Commons Pool Issue Type: Improvement Reporter: Phil Steitz When object pools determine that pooled objects need to be destroyed, they should provide context information to the factory indicating why the object is being destroyed. For example, a factory may want to dispose of abandoned objects differently than objects that have been invalidated or have been idle too long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (DBCP-559) 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效
[ https://issues.apache.org/jira/browse/DBCP-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173949#comment-17173949 ] Phil Steitz edited comment on DBCP-559 at 8/9/20, 6:44 PM: --- PR is [here|https://github.com/apache/commons-dbcp/pull/50]. * One thing missing in the list above, but made clear in javadoc is that connections checked out when restart is invoked don't count in the "new" pool's metrics. So for example if there are 3 out when restart or close - start is called, there may be as many as maxTotal + 3 out until the old ones get returned. It would be a little complicated to try to keep those connections in the post-restart counts. * I included no-op defaults for start (called "open" above) and restart in the interface spec to preserve backward compatibility. Any better ideas on how to handle that welcome. * I did not include eviction timer leak tests or much else to verify no side effects. More tests would be good. * Checkstyle now says BDS is too big. was (Author: psteitz): PR is [here|https://github.com/apache/commons-dbcp/pull/50]. * One thing missing in the list above, but made clear in javadoc is that connections checked out when restart is invoked don't count in the "new" pool's metrics. So for example if there are 3 out when restart or close - start is called, they may be as many as maxTotal + 3 out until the old ones get returned. It would be a little complicated to try to keep those connections in the post-restart counts. * I included no-op defaults for start (called "open" above) and restart in the interface spec to preserve backward compatibility. Any better ideas on how to handle that welcome. * I did not include eviction timer leak tests or much else to verify no side effects. More tests would be good. * Checkstyle now says BDS is too big. > 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效 > --- > > Key: DBCP-559 > URL: https://issues.apache.org/jira/browse/DBCP-559 > Project: Commons DBCP > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Wenzhi Ji >Priority: Major > Labels: ready-to-commit > Fix For: 2.8.0 > > Attachments: BasicDataSource.java, BasicDataSourceMXBean.java > > Original Estimate: 1h > Time Spent: 10m > Remaining Estimate: 50m > > 当前该开源软件具备的功能：若数据库连接被关闭（即closed置为true），则必须重启服务，才能使新的数据库连接生效（closed置为false）。 > 针对容灾场景，若数据库连接串发生变化，为了使新的数据库连接动态生效（热生效），不需要手工重启服务，建议该开源软件BasicDataSource类提供一个open()方法用于置closed > = false；以使建立的数据库连接生效。 > 变更的代码见附件。 > 注：附件是基于2.7.0版本代码进行优化的。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DBCP-559) 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效
[ https://issues.apache.org/jira/browse/DBCP-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173949#comment-17173949 ] Phil Steitz commented on DBCP-559: -- PR is [here|https://github.com/apache/commons-dbcp/pull/50]. * One thing missing in the list above, but made clear in javadoc is that connections checked out when restart is invoked don't count in the "new" pool's metrics. So for example if there are 3 out when restart or close - start is called, they may be as many as maxTotal + 3 out until the old ones get returned. It would be a little complicated to try to keep those connections in the post-restart counts. * I included no-op defaults for start (called "open" above) and restart in the interface spec to preserve backward compatibility. Any better ideas on how to handle that welcome. * I did not include eviction timer leak tests or much else to verify no side effects. More tests would be good. * Checkstyle now says BDS is too big. > 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效 > --- > > Key: DBCP-559 > URL: https://issues.apache.org/jira/browse/DBCP-559 > Project: Commons DBCP > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Wenzhi Ji >Priority: Major > Labels: ready-to-commit > Fix For: 2.8.0 > > Attachments: BasicDataSource.java, BasicDataSourceMXBean.java > > Original Estimate: 1h > Time Spent: 10m > Remaining Estimate: 50m > > 当前该开源软件具备的功能：若数据库连接被关闭（即closed置为true），则必须重启服务，才能使新的数据库连接生效（closed置为false）。 > 针对容灾场景，若数据库连接串发生变化，为了使新的数据库连接动态生效（热生效），不需要手工重启服务，建议该开源软件BasicDataSource类提供一个open()方法用于置closed > = false；以使建立的数据库连接生效。 > 变更的代码见附件。 > 注：附件是基于2.7.0版本代码进行优化的。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (DBCP-559) 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效
[ https://issues.apache.org/jira/browse/DBCP-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173447#comment-17173447 ] Phil Steitz edited comment on DBCP-559 at 8/8/20, 5:20 PM: --- I am working on a PR with tests for this, though I will happily review any PRs that come in before I finish. A [similar request|https://s.apache.org/gb0lk] was recently posted to tomcat-user. I am hoping that close followed by the simple open proposed in the attachment file (possibly also exposed as combined, "restart") will lead to # Idle connections get immediately closed (close contract) # Checked out connections get closed as they return (but abandoned connection cleanup stops for the pool being closed) # Any config changes made since initialization of running pool are applied to the new pool # New connections get sourced from a new pool that is initialized in "open" # No timer thread leaks or other untoward side effects I will try to add tests to confirm. was (Author: psteitz): I am working on a PR with tests for this, though I will happily review any PRs that come in before I finish. A [similar request|[https://lists.apache.org/thread.html/rf9006a5e3285bc32f0742ad4833c39852601305d5e56fc03dde60043%40%3Cusers.tomcat.apache.org%3E]] was recently posted to tomcat-user. I am hoping that close followed by the simple open proposed in the attachment file (possibly also exposed as combined, "restart") will lead to # Idle connections get immediately closed (close contract) # Checked out connections get closed as they return (but abandoned connection cleanup stops for the pool being closed) # Any config changes made since initialization of running pool are applied to the new pool # New connections get sourced from a new pool that is initialized in "open" # No timer thread leaks or other untoward side effects I will try to add tests to confirm. > 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效 > --- > > Key: DBCP-559 > URL: https://issues.apache.org/jira/browse/DBCP-559 > Project: Commons DBCP > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Wenzhi Ji >Priority: Major > Labels: ready-to-commit > Fix For: 2.8.0 > > Attachments: BasicDataSource.java, BasicDataSourceMXBean.java > > Original Estimate: 1h > Remaining Estimate: 1h > > 当前该开源软件具备的功能：若数据库连接被关闭（即closed置为true），则必须重启服务，才能使新的数据库连接生效（closed置为false）。 > 针对容灾场景，若数据库连接串发生变化，为了使新的数据库连接动态生效（热生效），不需要手工重启服务，建议该开源软件BasicDataSource类提供一个open()方法用于置closed > = false；以使建立的数据库连接生效。 > 变更的代码见附件。 > 注：附件是基于2.7.0版本代码进行优化的。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (DBCP-559) 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效
[ https://issues.apache.org/jira/browse/DBCP-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173447#comment-17173447 ] Phil Steitz edited comment on DBCP-559 at 8/8/20, 5:15 PM: --- I am working on a PR with tests for this, though I will happily review any PRs that come in before I finish. A [similar request|[https://lists.apache.org/thread.html/rf9006a5e3285bc32f0742ad4833c39852601305d5e56fc03dde60043%40%3Cusers.tomcat.apache.org%3E]] was recently posted to tomcat-user. I am hoping that close followed by the simple open proposed in the attachment file (possibly also exposed as combined, "restart") will lead to # Idle connections get immediately closed (close contract) # Checked out connections get closed as they return (but abandoned connection cleanup stops for the pool being closed) # Any config changes made since initialization of running pool are applied to the new pool # New connections get sourced from a new pool that is initialized in "open" # No timer thread leaks or other untoward side effects I will try to add tests to confirm. was (Author: psteitz): I am working on a PR with tests for this, though I will happily review any PRs that come in before I finish. A [similar request|[https://lists.apache.org/thread.html/rf9006a5e3285bc32f0742ad4833c39852601305d5e56fc03dde60043%40%3Cusers.tomcat.apache.org%3E]] was recently posted to tomcat-user. I am hoping that close followed by the simple open proposed in the attachment file (possibly also exposed as combined, "restart") will lead to # Idle connections get immediately closed (close contract) # Checked out connections get closed as they return (but abandoned connection cleanup stops for the pool being closed) # Any config changes made since initialization of running pool are applied to the new pool # New connections get sourced from a new pool that is initialized in "open" # No timer thread leaks or other untoward side effects I will try to add tests to confirm. > 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效 > --- > > Key: DBCP-559 > URL: https://issues.apache.org/jira/browse/DBCP-559 > Project: Commons DBCP > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Wenzhi Ji >Priority: Major > Labels: ready-to-commit > Fix For: 2.8.0 > > Attachments: BasicDataSource.java, BasicDataSourceMXBean.java > > Original Estimate: 1h > Remaining Estimate: 1h > > 当前该开源软件具备的功能：若数据库连接被关闭（即closed置为true），则必须重启服务，才能使新的数据库连接生效（closed置为false）。 > 针对容灾场景，若数据库连接串发生变化，为了使新的数据库连接动态生效（热生效），不需要手工重启服务，建议该开源软件BasicDataSource类提供一个open()方法用于置closed > = false；以使建立的数据库连接生效。 > 变更的代码见附件。 > 注：附件是基于2.7.0版本代码进行优化的。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DBCP-559) 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效
[ https://issues.apache.org/jira/browse/DBCP-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173447#comment-17173447 ] Phil Steitz commented on DBCP-559: -- I am working on a PR with tests for this, though I will happily review any PRs that come in before I finish. A [similar request|[https://lists.apache.org/thread.html/rf9006a5e3285bc32f0742ad4833c39852601305d5e56fc03dde60043%40%3Cusers.tomcat.apache.org%3E]] was recently posted to tomcat-user. I am hoping that close followed by the simple open proposed in the attachment file (possibly also exposed as combined, "restart") will lead to # Idle connections get immediately closed (close contract) # Checked out connections get closed as they return (but abandoned connection cleanup stops for the pool being closed) # Any config changes made since initialization of running pool are applied to the new pool # New connections get sourced from a new pool that is initialized in "open" # No timer thread leaks or other untoward side effects I will try to add tests to confirm. > 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效 > --- > > Key: DBCP-559 > URL: https://issues.apache.org/jira/browse/DBCP-559 > Project: Commons DBCP > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Wenzhi Ji >Priority: Major > Labels: ready-to-commit > Fix For: 2.8.0 > > Attachments: BasicDataSource.java, BasicDataSourceMXBean.java > > Original Estimate: 1h > Remaining Estimate: 1h > > 当前该开源软件具备的功能：若数据库连接被关闭（即closed置为true），则必须重启服务，才能使新的数据库连接生效（closed置为false）。 > 针对容灾场景，若数据库连接串发生变化，为了使新的数据库连接动态生效（热生效），不需要手工重启服务，建议该开源软件BasicDataSource类提供一个open()方法用于置closed > = false；以使建立的数据库连接生效。 > 变更的代码见附件。 > 注：附件是基于2.7.0版本代码进行优化的。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (DBCP-563) query return previous result when previous request cause MySQLTimeoutException by one connection
[
https://issues.apache.org/jira/browse/DBCP-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Steitz closed DBCP-563.
Resolution: Not A Problem
This is a driver bug. I can reproduce using mysql-connector-java-5.x but not
with mysql-connector-java-8. DBCP can't change connections under the handles
held by clients and even that would not explain the result. See
[https://bugs.mysql.com/bug.php?id=99495]
> query return previous result when previous request cause
> MySQLTimeoutException by one connection
>
>
> Key: DBCP-563
> URL: https://issues.apache.org/jira/browse/DBCP-563
> Project: Commons DBCP
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: jode
>Priority: Major
>
> when run JdbcLocalDemo, you can get wrong result。query a record，then return b
> record。
> {code:java}
> import org.apache.commons.dbcp2.BasicDataSource;
> import org.apache.commons.dbcp2.BasicDataSourceFactory;
> import java.sql.Connection;
> import java.sql.PreparedStatement;
> import java.sql.ResultSet;
> import java.sql.SQLException;
> import java.util.concurrent.ExecutorService;
> import java.util.concurrent.Executors;
> import java.util.concurrent.ThreadLocalRandom;
> import java.util.concurrent.TimeUnit;
> import java.util.concurrent.atomic.AtomicInteger;
> import static java.sql.ResultSet.CLOSE_CURSORS_AT_COMMIT;
> import static java.sql.ResultSet.CONCUR_READ_ONLY;
> import static java.sql.ResultSet.TYPE_FORWARD_ONLY;
> public class JdbcLocalDemo {
> private static final AtomicInteger count = new AtomicInteger(0);
> static BasicDataSource basicDataSource = null;
> static {
> try {
> basicDataSource =
> BasicDataSourceFactory.createDataSource(PropertiesUtils.loadFromClassPath("application-local.properties"));
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> public static void main(String[] args) throws Exception {
> SqlRunnable r1 = new SqlRunnable("a");
> SqlRunnable r2 = new SqlRunnable("b");
> ExecutorService executorService = Executors.newFixedThreadPool(5);
> for(int i = 1; i <= 2000; ++i) {
> executorService.submit(r1);
> executorService.submit(r2);
> }
> executorService.shutdown();
> executorService.awaitTermination(1, TimeUnit.MINUTES);
> }
> private static class SqlRunnable implements Runnable {
> private final String value;
> private SqlRunnable(String value) {
> this.value = value;
> }
> @Override
> public void run() {
> PreparedStatement preparedStatement = null;
> ResultSet resultSet = null;
> Connection connection = null;
> System.out.println(count.incrementAndGet());
> try {
> connection = basicDataSource.getConnection();
> preparedStatement = connection.prepareStatement("select \"" +
> value + "\" as v, sleep(?);", TYPE_FORWARD_ONLY, CONCUR_READ_ONLY,
> CLOSE_CURSORS_AT_COMMIT);
> preparedStatement.setQueryTimeout(1);
> preparedStatement.setFetchSize(0);
> preparedStatement.setMaxRows(0);
> //preparedStatement.setString(1, value);
>
> preparedStatement.setDouble(1,ThreadLocalRandom.current().nextInt(2000) /
> 1000.0);
> resultSet = preparedStatement.executeQuery();
> while (resultSet.next()) {
> String v = resultSet.getString("v");
> if (!value.equals(v)) {
> System.err.println("wrong result
> !=!=!=!=!=!=!=!=!=!=!=!=!=!=!=!=!=!=!=!=!=: " + value);
> System.exit(-1);
> }
> }
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> if (resultSet != null) {
> try {
> resultSet.close();
> } catch (SQLException e) {
> e.printStackTrace();
> }
> }
> if (preparedStatement != null) {
> try {
> preparedStatement.close();
> } catch (SQLException e) {
> e.printStackTrace();
> }
> }
> if (connection != null) {
> try {
> connection.close();
> } catch (SQLException e) {
> e.printStackTrace();
> }
> }
> }
> }
> }
> }
> {code}
> {code:java}
> //
[jira] [Commented] (DBCP-559) 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效
[ https://issues.apache.org/jira/browse/DBCP-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166644#comment-17166644 ] Phil Steitz commented on DBCP-559: -- This looks reasonable to me. The change is just to add an "open" method to BDS that sets closed to false, which will effectively cause the datasource to re-initialize itself when getConnection is called. The change to the JMX class is just to expose the method. Seems a safe and useful change to me, unless I am missing something. It might be best to change the name to "start" and maybe also expose "stop" and "restart" via JMX. Does anyone see any reason not to add these methods? > 数据库连接可以通过BasicDataSource类的close()方法进行关闭，但该类没有提供open()方法使得新建的数据库连接生效 > --- > > Key: DBCP-559 > URL: https://issues.apache.org/jira/browse/DBCP-559 > Project: Commons DBCP > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Wenzhi Ji >Priority: Major > Labels: ready-to-commit > Fix For: 2.8.0 > > Attachments: BasicDataSource.java, BasicDataSourceMXBean.java > > Original Estimate: 1h > Remaining Estimate: 1h > > 当前该开源软件具备的功能：若数据库连接被关闭（即closed置为true），则必须重启服务，才能使新的数据库连接生效（closed置为false）。 > 针对容灾场景，若数据库连接串发生变化，为了使新的数据库连接动态生效（热生效），不需要手工重启服务，建议该开源软件BasicDataSource类提供一个open()方法用于置closed > = false；以使建立的数据库连接生效。 > 变更的代码见附件。 > 注：附件是基于2.7.0版本代码进行优化的。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (POOL-386) Closing a pool can cause Evictor in another pool to be cancelled
[
https://issues.apache.org/jira/browse/POOL-386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162100#comment-17162100
]
Phil Steitz edited comment on POOL-386 at 7/21/20, 6:04 PM:
A couple things to note / possibly improve in the patch:
# Usage tracking is accomplished by maintaining a hashmap whose keys are weak
references to eviction tasks belonging to client pools. A Reaper task has been
added to the executor that iterates the map and removes references (and tasks)
to tasks that have been gc-ed. When there are no live tasks left, it shuts
down the executor. The Reaper is created when a new executor is launched,
which is triggered by a schedule
call. The Reaper inherits the run frequency of the task being scheduled when
it is created. Not sure if this is the best, but seems reasonable.
# The executor thread pool size is not increased by the patch. It may make
sense to bump this to 2 or to make it configurable. AFAICT, concurrent
activation of evictor tasks should be OK, as they are guarded by the eviction
lock in BaseGenericObjectPool.
# The Reaper has to acquire the EvictionTimer class lock (like schedule and
cancel). I don't see contention for this lock as likely to cause problems
because schedule and cancel calls should in general only happen when starting /
stopping pools that run evictors, but in theory contention for this lock could
cause the Reaper to hold up the other tasks. This is another reason to
consider adding another thread to the executor pool or even running the Reaper
under a separate executor.
was (Author: psteitz):
A couple things to note / possibly improve in the patch:
# Usage tracking is accomplished by maintaining a hashmap whose keys are weak
references to eviction tasks belonging to client pools. A Reaper task has been
added to the executor that iterates the map and removes references (and tasks)
to tasks that have been gc-ed. When there are no live tasks left, it shuts
down the executor. The Reaper is created when a new executor is launched,
which is triggered by a schedule
call. The Reaper inherits the run frequency of the task being scheduled when
it is created. Not sure if this is the best, but seems reasonable.
# The executor thread pool size is not increased by the patch. It may make
sense to bump this to 2 or to make it configurable. AFAICT, concurrent
activation of evictor tasks should be OK, as they are guarded by the eviction
lock in BaseGenericObjectPool.
> Closing a pool can cause Evictor in another pool to be cancelled
>
>
> Key: POOL-386
> URL: https://issues.apache.org/jira/browse/POOL-386
> Project: Commons Pool
> Issue Type: Task
>Affects Versions: 2.6.1, 2.6.2, 2.7.0, 2.8.0
>Reporter: Phil Steitz
>Priority: Major
> Fix For: 2.8.1
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The fix for POOL-337 introduced a race condition that can cause the shared
> EvictionTimer to be cancelled when a pool is closed but another pool still
> has an active Evictor. The EvictionTimer cancel method used to cancel an
> eviction task owned by a client pool uses this test to determine whether or
> not to shutdown its executor:
> {code:java}
> if (executor != null && executor.getQueue().isEmpty()){code}
> The executor may report an empty queue if it is executing a task. This will
> cause the executor to be shut down and scheduling of new tasks to stop.
> The unit test below illustrates the problem.
> {code:java}
>
> public void testEvictionTimerMultiplePools() throws InterruptedException {
> final AtomicIntegerFactory factory = new AtomicIntegerFactory();
> factory.setValidateLatency(50);
> final GenericObjectPool evictingPool = new
> GenericObjectPool<>(factory);
> evictingPool.setTimeBetweenEvictionRunsMillis(100);
> evictingPool.setNumTestsPerEvictionRun(5);
> evictingPool.setTestWhileIdle(true);
> evictingPool.setMinEvictableIdleTimeMillis(50);
> for (int i = 0; i < 10; i++) {
> try {
> evictingPool.addObject();
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> for (int i = 0; i < 1000; i++) {
> GenericObjectPool nonEvictingPool = new
> GenericObjectPool<>(factory);
> nonEvictingPool.close();
> }
> Thread.sleep(1000);
> Assert.assertEquals(0, evictingPool.getNumIdle());
> evictingPool.close();
> }
> {code}
> Proposed fix:
> Change the test guarding executor shutdown to
> {code:java}
> executor.getQueue().isEmpty() && executor.getActiveCount() == 0
> {code}
> This still exposes a low-probability race - a task completes after the
>
