[ANNOUNCEMENT] Apache Commons Pool 2.12.0

[Phil Steitz]

The Apache Commons team is pleased to announce the release of
Apache Commons Pool 2.12.0.

Commons Pool provides an object-pooling API and a number of object pool
implementations.

Java 8 or above is required.

Version 2.12.0 is a maintenance release, including bug fixes and
enhancements.  This release is source and binary compatible with the prior
version, 2.11.1.  All users are encouraged to upgrade.

A complete historical list of changes to Commons Pool is available here:
https://commons.apache.org/proper/commons-pool/changes-report.html

For complete information on Apache Commons Pool, including instructions on
how to submit bug reports, patches, or suggestions for improvement, see the
Apache Commons Pool website:
https://commons.apache.org/proper/commons-pool/

Source and binary distributions are available from the Commons Pool
download page:
https://commons.apache.org/proper/commons-pool/download_pool.cgi

Please verify the hashes and signatures on downloaded artifacts.

The 2.x versions of Commons Pool are distributed under the commons-pool2
maven artifactId. To pull Commons Pool 2.12.0 into an Apache Maven build as
a dependency, use


org.apache.commons
commons-pool2
2.12.0


Thanks in advance for bug reports, suggestions for improvement, patches or
other contributions to the Apache Commons community.

Phil Steitz
-Apache Commons Team

Re: Updating Apache Commons DBCP 2.10.0

[Phil Steitz]

On Tue, Sep 26, 2023 at 4:50 AM Gary Gregory  wrote:

> (Let's stop high jacking the announcement thread.)
>

Agreed.  Sorry about that.

>
> In the future, FYI, I plan on supporting both Java and Jakarta by splitting
> DBCP into a multi-module Maven project. I'm not sure exactly when but it's
> on my to do list.
>

That sounds good.  But we may need to do something before that to restore
backward compatibility (if there is in fact a problem).  The build error
that Greg reports below is somehow related to the Commons Parent 62
version.  If I back rev Commons Parent to version 60 and revert the JTA
spec jar change (per below), the build works.  (Greg - can you try that -
just change Commons Parent version to 60 along with the change below).

Alternatively, maybe there is some other workaround to avoid the conflict
that Greg is reporting.  I am not sure if there is in fact a backward
compat issue here.  Any better ideas?

One more thing is the EPL/GPL license on the Jakarta jars.   Have these
been cleared for use (as dependencies) in ASF projects? Do we need some
kind of notice for this?

Phil

>
> Gary
>
>
>
> On Tue, Sep 26, 2023, 5:00 AM Greg Huber  wrote:
>
> > If I am understanding this correctly, I replace
> >
> > 
> > 
> >   jakarta.transaction
> >   jakarta.transaction-api
> >   1.3.1
> > 
> >
> > with
> >
> > 
> > org.apache.geronimo.specs
> > geronimo-jta_1.1_spec
> > 1.1.1
> > true
> > 
> >
> > in the source of 2.10.0 pom.xml and then mvn clean package.
> >
> > I get this error
> >
> > [ERROR] Failed to execute goal
> > org.moditect:moditect-maven-plugin:1.0.0.Final:add-module-info
> > (add-module-infos) on project commons-dbcp2: Execution add-module-infos
> > of goal org.moditect:moditect-maven-plugin:1.0.0.Final:add-module-info
> > failed: No assignedModuleName provided for jar with invalid module name:
> >
> ~/.m2/repository/org/apache/geronimo/specs/geronimo-jta_1.1_spec/1.1.1/geronimo-jta_1.1_spec-1.1.1.jar
> >
> > -> [Help 1]
> >
> > #
> >
> >
> > I did a diff of my apps /lib folder between the 2.10.0 and 2.9.0
> >
> > Only in ~/downloads/commons-dbcp2-2.10.0-src/target/lib.9:
> > commons-dbcp2-2.9.0.jar
> > Only in ~/downloads/commons-dbcp2-2.10.0-src/target/lib.10:
> > commons-dbcp2-2.10.0.jar
> >
> > Only in ~/downloads/commons-dbcp2-2.10.0-src/target/lib.9:
> > commons-pool2-2.10.0.jar
> > Only in ~/downloads/commons-dbcp2-2.10.0-src/target/lib.10:
> > commons-pool2-2.11.1.jar
> >
> > Only in ~/downloads/commons-dbcp2-2.10.0-src/target/lib.9:
> > jakarta.transaction-api-2.0.1.jar
> > Only in ~/downloads/commons-dbcp2-2.10.0-src/target/lib.10:
> > jakarta.transaction-api-1.3.1.jar
> >
> >
> > If I copy lib.9: jakarta.transaction-api-2.0.1.jar to replace the
> > lib.10: jakarta.transaction-api-1.3.1.jar class
> > ThreadLocalSessionContextNoAutoClose compiles and works.
> >
> > Cheers Greg
> >
> > On 25/09/2023 18:49, Phil Steitz wrote:
> > > Thanks, Greg.  I assume that dependency was there and working with DBCP
> > > 2.9, correct?
> > >
> > > Can you try reverting the pom change above, replacing the
> > > Jakarta transactions reference with
> > >
> > > 
> > >org.apache.geronimo.specs
> > >geronimo-jta_1.1_spec
> > >1.1.1
> > >true
> > > 
> > >
> > > Then recompile DBCP from 2.10 sources and test (just use mvn clean
> > package
> > > from the root of an extract of the source distribution).
> > >
> > > Phil
> > >
> > > On Mon, Sep 25, 2023 at 12:44 AM Greg Huber 
> wrote:
> > >
> > >>>> does your code maybe bring in the 2.0 jakarta spec jar?
> > >>
> > >> I do have :
> > >>
> > >> jakarta.transaction-api-2.0.1.jar
> > >>
> > >> which comes from hibernate-core 6.3.1.Final.
> > >>
> > >>
> > >>
> > >> ###
> > >>
> > >> Using eclipse here are the errors
> > >>
> > >> The project was not built since its build path is incomplete. Cannot
> > >> find the class file for jakarta.transaction.Synchronization. Fix the
> > >> build path then try building this project
> > >>
> > >> The type jakarta.transaction.Synchronization cannot be resolved. It is
> > >> indirectly referen

Re: [ANNOUNCEMENT] Apache Commons DBCP 2.10.0

[Phil Steitz]

Thanks, Greg.  I assume that dependency was there and working with DBCP
2.9, correct?

Can you try reverting the pom change above, replacing the
Jakarta transactions reference with


  org.apache.geronimo.specs
  geronimo-jta_1.1_spec
  1.1.1
  true


Then recompile DBCP from 2.10 sources and test (just use mvn clean package
from the root of an extract of the source distribution).

Phil

On Mon, Sep 25, 2023 at 12:44 AM Greg Huber  wrote:

>   >> does your code maybe bring in the 2.0 jakarta spec jar?
>
> I do have :
>
> jakarta.transaction-api-2.0.1.jar
>
> which comes from hibernate-core 6.3.1.Final.
>
>
>
> ###
>
> Using eclipse here are the errors
>
> The project was not built since its build path is incomplete. Cannot
> find the class file for jakarta.transaction.Synchronization. Fix the
> build path then try building this project
>
> The type jakarta.transaction.Synchronization cannot be resolved. It is
> indirectly referenced from required type
> org.hibernate.context.internal.ThreadLocalSessionContext.CleanupSync
>
>
> Here is the class that won't compile
>
> public class ThreadLocalSessionContextNoAutoClose
>  extends ThreadLocalSessionContext {
>
>  private static final long serialVersionUID = -9220338553393731611L;
>
>  /**
>   * Create a new instance.
>   *
>   * @param factory The SessionFactoryImplementor
> required by the
>   *super constructor.
>   */
>  public ThreadLocalSessionContextNoAutoClose(
>  SessionFactoryImplementor factory) {
>  super(factory);
>  }
>
>  /**
>   * Returns false to prevent auto closing.
>   *
>   * @return false to prevent auto closing.
>   */
>  protected boolean isAutoCloseEnabled() {
>  return false;
>  }
>
>  /**
>   * Returns false to prevent auto flushing.
>   *
>   * @return false to prevent auto flushing.
>   */
>  protected boolean isAutoFlushEnabled() {
>  return false;
>  }
>
>  /**
>   * Uses super.buildOrObtainSession(), then sets the
> resulting
>   * Session's flush mode to FlushMode.NEVER
> to
>   * prevent auto-flushing.
>   *
>   * @return A session configured with FlushMode.NEVER.
>   */
>  protected Session buildOrObtainSession() {
>  Session s = super.buildOrObtainSession();
>  s.setHibernateFlushMode(FlushMode.MANUAL);
>  return s;
>  }
>
>  /**
>   * Returns an instance of CleanupSynch which prevents
> auto
>   * closing and unbinding.
>   *
>   * @return A CleanupSynch which prevents auto closing and
>   * unbinding.
>   */
>  protected CleanupSync buildCleanupSynch() {
>  return new NoCleanupSynch(factory());
>  }
>
>  /**
>   * A simple extension of CleanupSynch that prevents
> any cleanup
>   * from happening. No session closing or unbinding.
>   */
>  private static class NoCleanupSynch
>  extends ThreadLocalSessionContext.CleanupSync {
>
>  private static final long serialVersionUID =
> -6191453375299821467L;
>
>  /**
>   * Creates a new instance based on the given factory.
>   *
>   * @param factory The required SessionFactory that is
>   *passed to the super constructor.
>   */
>  public NoCleanupSynch(SessionFactory factory) {
>  super(factory);
>  }
>
>  /**
>   * Does nothing, thus helping to prevent session closing and/or
>   * unbinding.
>   */
>  public void beforeCompletion() {
>  // do nothing
>  }
>
>  /**
>   * Does nothing, thus helping to prevent session closing and/or
>   * unbinding.
>   *
>   * @param i
>   */
>  public void afterCompletion(int i) {
>  // do nothing
>  }
>  }
>
> }
>
> Thanks.
>
> On 24/09/2023 20:34, Phil Steitz wrote:
> > On Sun, Sep 24, 2023 at 10:28 AM Greg Huber  wrote:
> >
> >> Forgot to add, I use hibernate 6.
> >>
> > I suspect this is a side effect of the change to use the jakarta spec jar
> >
> > In 2.10, we have
> >
> > 
> >  jakarta.transaction
> >  jakarta.transaction-api
> >  1.3.1
> > 
> >
> > vs
> >
> > 
> >org.apache.geronimo.specs
> >geronimo-jta_1.1_spec
> >1.1.1
> &

Re: [ANNOUNCEMENT] Apache Commons DBCP 2.10.0

[Phil Steitz]

On Sun, Sep 24, 2023 at 10:28 AM Greg Huber  wrote:

> Forgot to add, I use hibernate 6.
>

I suspect this is a side effect of the change to use the jakarta spec jar

In 2.10, we have


jakarta.transaction
jakarta.transaction-api
1.3.1


vs


  org.apache.geronimo.specs
  geronimo-jta_1.1_spec
  1.1.1
  true


in 2.9.  What I don't get is where Greg's code is picking
up jakarta.transaction.Synchronization because the jakarta version we
"upgraded" to still exports the javax classes and these are what dbcp
uses.  Greg - does your code maybe bring in the 2.0 jakarta spec jar?

Phil


>
> Cheers.
>
> On Sun, 24 Sept 2023 at 12:49, Gary Gregory 
> wrote:
>
> > What else has changed in your app?
> >
> > Gary
> >
> > On Sun, Sep 24, 2023 at 5:13 AM Greg Huber  wrote:
> > >
> > > Hello,
> > >
> > > On upgrading from 2.9.0 I get this message and my class won't compile.
> > >
> > > I get this error
> > >
> > > The type jakarta.transaction.Synchronization cannot be resolved. It is
> > > indirectly referenced from required type
> > > org.hibernate.context.internal.ThreadLocalSessionContext.CleanupSync
> > >
> > > I am on :
> > >
> > > openjdk version "11.0.19" 2023-04-18 LTS
> > > OpenJDK Runtime Environment Zulu11.64+19-CA (build 11.0.19+7-LTS)
> > > OpenJDK 64-Bit Server VM Zulu11.64+19-CA (build 11.0.19+7-LTS, mixed
> > mode)
> > >
> > > Not too sure what I need to do to fix this.
> > >
> > > Cheers
> > >
> > > On 03/09/2023 22:51, Gary Gregory wrote:
> > > > The Apache Commons DBCP team is pleased to announce the release of
> > > > Apache Commons DBCP 2.10.0.
> > > >
> > > > Apache Commons DBCP software implements Database Connection Pooling.
> > > >
> > > > This is a minor release, including bug fixes and enhancements:
> > > >
> >
> https://commons.apache.org/proper/commons-dbcp/changes-report.html#a2.10.0
> > > >
> > > > For complete information on Apache Commons DBCP, including
> > > > instructions on how to submit bug reports, patches, or suggestions
> for
> > > > improvement, see the Apache Commons DBCP website:
> > > >
> > > > https://commons.apache.org/dbcp/
> > > >
> > > > Download page: https://commons.apache.org/dbcp/download_dbcp.cgi
> > > >
> > > > Gary Gregory
> > > > Apache Commons
> > > >
> > > > -
> > > > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> > > > For additional commands, e-mail: user-h...@commons.apache.org
> > > >
> > >
> > > -
> > > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> > > For additional commands, e-mail: user-h...@commons.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> > For additional commands, e-mail: user-h...@commons.apache.org
> >
> >
>

Re: [IO] Java's InflaterInputStream not playing well with IOUtils.skipFully

[Phil Steitz]

On Mon, Jun 12, 2023 at 6:52 PM Phil Steitz  wrote:

>
>
> On Mon, Jun 12, 2023 at 2:14 PM Tim Allison  wrote:
>
>> All,
>>
>>   Since the refactoring to avoid threadlocal in skipFully, we are now
>> getting errors in one of our unit tests on Tika [0].  We're still on
>> commons-io 2.11.0, but we found this problem with 2.12.0 and 2.13.0.
>>
>>I was able to reproduce the problem outside of Tika [1].  To reproduce
>> the problem, the stream has to be wrapped in java's InflaterInputStream,
>> and it has to be run multithreaded.  The problem is easily reproducible
>> with only two threads, but there is variation on which iteration triggers
>> the problem even with the same seed (what you'd expect from a
>> multi-threading bug).
>>
>>   I don't think this is a bug in commons-io, per se, but it is mildly
>> frightening from a data reliability perspective that the combination of
>> IOUtils.skipFully on top of Java's InflaterInputStream can lead to
>> different data.
>>
>>   Is this a bug in Java, a bug in commons-io or something else?  What
>> should we do?  Thank you!
>>
>
> Interesting.  Taking a quick look at the source for both IOUtils and
> InflaterInputStream, it looks to me like the InflaterInputStream may be
> effectively assuming that it has exclusive access to the buffer that it is
> reading into.   I looked at a couple of jdk sources and they both look like
> this:
>
> public int read(byte[] b, int off, int len) throws IOException {
> ensureOpen();
> if (b == null) {
> throw new NullPointerException();
> } else if (off < 0 || len < 0 || len > b.length - off) {
> throw new IndexOutOfBoundsException();
> } else if (len == 0) {
> return 0;
> }
> try {
> int n;
> while ((n = inf.inflate(b, off, len)) == 0) {
> if (inf.finished() || inf.needsDictionary()) {
> reachEOF = true;
> return -1;
> }
> if (inf.needsInput()) {
> fill();
> }
> }
> return n;
> } catch (DataFormatException e) {
> String s = e.getMessage();
> throw new ZipException(s != null ? s : "Invalid ZLIB data
> format");
> }
> }
>
> I can see how zero-ing the byte array it is reading to could cause
> problems here.  I don't see any doc anywhere saying that either this Reader
> or the interface in general has this constraint, so maybe it is a jdk bug,
> but it is what it is and I think to be safe it would be better to add back
> the ThreadLocal on the write only bufffer in IOUtills or change it to not
> zero the array.  I don't see why the zeroing is necessary, but I don't know
> much about [io].
>

Sorry, just eliminating the zero-ing probably wont work because concurrent
writes could make buffer segments not inflatable IIUC what is going on.  So
I would just restore the protection.

>
> Phil
>
>
>
>>Best,
>>
>>Tim
>>
>>
>>
>>
>> [0] https://issues.apache.org/jira/browse/TIKA-4065
>> [1]
>>
>> https://github.com/tballison/commons-io/blob/TIKA-4065/src/test/java/org/apache/commons/io/IOUtilsMultithreadedTest.java
>>
>

Re: [IO] Java's InflaterInputStream not playing well with IOUtils.skipFully

[Phil Steitz]

On Mon, Jun 12, 2023 at 2:14 PM Tim Allison  wrote:

> All,
>
>   Since the refactoring to avoid threadlocal in skipFully, we are now
> getting errors in one of our unit tests on Tika [0].  We're still on
> commons-io 2.11.0, but we found this problem with 2.12.0 and 2.13.0.
>
>I was able to reproduce the problem outside of Tika [1].  To reproduce
> the problem, the stream has to be wrapped in java's InflaterInputStream,
> and it has to be run multithreaded.  The problem is easily reproducible
> with only two threads, but there is variation on which iteration triggers
> the problem even with the same seed (what you'd expect from a
> multi-threading bug).
>
>   I don't think this is a bug in commons-io, per se, but it is mildly
> frightening from a data reliability perspective that the combination of
> IOUtils.skipFully on top of Java's InflaterInputStream can lead to
> different data.
>
>   Is this a bug in Java, a bug in commons-io or something else?  What
> should we do?  Thank you!
>

Interesting.  Taking a quick look at the source for both IOUtils and
InflaterInputStream, it looks to me like the InflaterInputStream may be
effectively assuming that it has exclusive access to the buffer that it is
reading into.   I looked at a couple of jdk sources and they both look like
this:

public int read(byte[] b, int off, int len) throws IOException {
ensureOpen();
if (b == null) {
throw new NullPointerException();
} else if (off < 0 || len < 0 || len > b.length - off) {
throw new IndexOutOfBoundsException();
} else if (len == 0) {
return 0;
}
try {
int n;
while ((n = inf.inflate(b, off, len)) == 0) {
if (inf.finished() || inf.needsDictionary()) {
reachEOF = true;
return -1;
}
if (inf.needsInput()) {
fill();
}
}
return n;
} catch (DataFormatException e) {
String s = e.getMessage();
throw new ZipException(s != null ? s : "Invalid ZLIB data
format");
}
}

I can see how zero-ing the byte array it is reading to could cause problems
here.  I don't see any doc anywhere saying that either this Reader or the
interface in general has this constraint, so maybe it is a jdk bug, but it
is what it is and I think to be safe it would be better to add back the
ThreadLocal on the write only bufffer in IOUtills or change it to not zero
the array.  I don't see why the zeroing is necessary, but I don't know much
about [io].

Phil



>Best,
>
>Tim
>
>
>
>
> [0] https://issues.apache.org/jira/browse/TIKA-4065
> [1]
>
> https://github.com/tballison/commons-io/blob/TIKA-4065/src/test/java/org/apache/commons/io/IOUtilsMultithreadedTest.java
>

Re: [dbcp2] No error/warning when returning a connection twice

[Phil Steitz]

On 4/14/21 12:59 PM, Gary Gregory wrote:

Hi,

That's a tough sell IMO because (1) the JDBC specification does not require
this behavior AND (2) it would break existing applications. What is the
harm is having nothing happen when you close a closed connection?


That was the thinking when we changed this to the current behavior back 
in 1.3.  See https://issues.apache.org/jira/browse/DBCP-233


Phil



What am I missing?

Gary


On Wed, Apr 14, 2021, 14:47 Christopher Schultz <
ch...@christopherschultz.net> wrote:


All,

I have had an application for years which I run in development with
maxsize=1 so I can catch any double-check-out errors as soon as they
might read their ugly heads.

But I never considered double-return logic errors.

I recently wrote a "single connection pool" which is bare-bones and just
creates a single JDBC connection and hands it out to anyone who wants
it. It counts borrows vs returns and it was vomiting whenever we called
a particular method which, evidently, has been double-closing
connections for years.

I had expected that DBCP2 would be objecting to double-closes, but
evidently not.

I checked, and there is no configuration that I can find to enable some
kind of error/notification when double-closes are performed.

Is that something the project would be interested in adding?

I'm talking about something like this:

Connection conn = dbcp.getConnection();

// do stuff

conn.close(); // great, returns to pool
conn.close(); // should bomb

Thanks,
-chris

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] Optimal defaults for DSpace

[Phil Steitz]

On 12/20/20 10:26 PM, Hrafn Malmquist wrote:

Hi Gary

Thanks for taking the time to respond.

I hope you can bear with me as I am still learning about database
connection pooling.

Perhaps I did not ask the question correctly. I am not asking about a site
specific setup but rather what defaults should be shipped with the
software. I am part of the minor version release team.

Currently, the default setup is a DBCP2 v. 2.1.1 connection pool with
only maxWaitMillis,
maxIdle and maxTotal configurable in the DSpace configuration settings and
the default values for these settings set to 5000, 10 and 30 respectively.
It's unclear why these defaults were chosen to begin with, git blame shows
they were chosen back in 2015. I don't think a lot of thought went into
choosing 1) which parameters should be configurable nor 2) what their
defaults should be (or why they should differ from DBCP2 defaults).

DSpace repositories are run by higher education institutions and all sorts
of institutions and organisations involved in research, for instance the
Smithsonian (https://repository.si.edu/). Therefore, although the vast
majority of instances are run by small institutions that get little
traffic, others are likely to receive relatively heavy traffic, from users
and crawlers.

So the idea is to ask the experts what parameters should be configurable
for the average repository admin, keeping in mind that the aim is for
installation and setup to be simple (in effect, what are the "main"
parameters likely to need tweaking) and what should the out-of-the-box
defaults be (if at all different from the DBCP2 defaults).

I am particularly surprised at the low maxWaitMillis chosen. Is that not
likely to cause problems for high traffic sites?


I would say no.  Having threads blocked waiting for connections for 
longer than 5 seconds will likely cause problems in heavily loaded 
applications.  You will end up running out of app server processing 
threads if they are hanging for that long.   If getConnection is taking 
that long, there is likely a problem somewhere in the overall system - 
processing threads holding connections too long, not enough connections, 
database latency, etc.  It all comes down to queuing theory.  If your 
app does not hold connections long and queries are optimized, even a 
relatively small pool can handle decent load.  The key is to not to 
leave connections open or hold on to them too long.


The defaults above look OK to me, though if database connections are not 
in short supply, I would bump maxIdle to 20.  The reason for this is 
that setting it at 10 means that if the number used regularly goes up to 
20+, you will end up with a lot of connection churn.  On the other hand, 
if the usage pattern is spikes now and then followed by long periods of 
lighter load, setting it at 20 will "waste" some connections.  How 
important that "waste" is depends on what else is going on in the DB, 
how many pools are sharing it, etc.


I would recommend upgrading to the latest version compatible with the 
version of tc you are running, or simply using the version that ships 
with tomcat (which is generally the latest compatible). Another reason 
to upgrade dbcp if you are using it directly is to pick up the fixes in 
the later version of commons pool that it brings in.


For some general info on how dbcp and pool configs work, see [1]. It is 
old, but the basic concepts are still correct.  If you are familiar with 
queuing theory, you can view a pool with n connections as a M/M/n 
queue.  What drives everything is request arrival rate and service time, 
which in the case of dbcp is how long an application thread holds a 
connection.   You can observe actual utilization using the JMX interfaces.


Phil

[1] https://www.slideshare.net/psteitz/apachecon2014-pooldbcp


Best regards, Hrafn


[1] :
https://github.com/DSpace/DSpace/blob/250c87dc1604c34e2a963b6804163c73278e9ff7/dspace/config/spring/api/core-hibernate.xml#L41-L48

[2] :
https://github.com/DSpace/DSpace/blob/250c87dc1604c34e2a963b6804163c73278e9ff7/dspace/config/dspace.cfg#L77-L86

On Sun, Dec 20, 2020 at 6:40 PM Gary Gregory  wrote:


Hi,

Each new DBCP release brings fixes, additions,  and other updates, as you
can read in the release notes.

How to best configure DBCP for any given combination of JDBC driver, its
database, and application will be quite variable, which is somewhat out of
scope here IMO.

Gary

On Fri, Dec 18, 2020, 11:15 Hrafn Malmquist 
wrote:


Good day

I'm wondering what are optimal defaults for DSpace, open source digital
repository software aimed especially at  academic, non-profit, and
commercial organizations (see https://duraspace.org/dspace/).

DSpace supports both Postgres and Oracle and recommends Tomcat, Jetty or
Caucho Resin. I suspect 9/10 installations use Tomcat.

DSpace comes packaged with Apache Commons DCBP 2.1.1. DSpace only
configures three configurations for DBCP2 using non-default settings.

(see:

[1] and [2])

Re: DB Connection Restore Issue with Apache Commons DBCP 2.5.0

[Phil Steitz]

What exactly do you mean by "restoring?"

Phil

On 10/30/19 11:17 PM, Shihad Salam wrote:

Hi Team,

Please help me in resolving the below issue:

While using the DBCP2 (2.5.0) connection pool in my SpringBoot project with
MS SQL Server, the database connection is not restoring after a DB service
restart. This is working fine when using the SpringBoot default Hikari
Connection Pool. Is this a bug with DBCP2?  If not, how can I solve this
issue?

I have used the below configurations:
**.dbcp2.test-while-idle=true*
**.dbcp2.test-on-borrow=true*
**.dbcp2.time-between-eviction-runs-millis=6*
**.dbcp2.validation-query=SELECT 1*
**.dbcp2.min-idle=1*
**.dbcp2.max-total=100*

**.dbcp2.max-idle=50*

P.S : If this mail is not intended to this mail group, please forward to
the respective team.

Thanks,
Shihad




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] in DBCP 1.4 - Any auto-reconnect property available ?

[Phil Steitz]

On 1/15/19 2:59 PM, Matt Sicker wrote:

One idea might be to invoke a crafted query for your connection check
that fails every so often so it'll be forced to reconnect, though I
haven't tried that before.


I was thinking the same thing.  I am not sure how to do it, but I 
vaguely recall it being possible to get information about an Oracle 
session via SQL query.  If your validation query could check when 
the connection was established it could fail if it was too old.  
Validation failures cause connections to be closed and subsequent 
borrows will cause new ones to be created.  You could set this up 
using either testOnBorrow or testOnReturn (or both).


Of course, the best is to upgrade to DBCP 2 where the connection 
lifetime property you want is directly available.


Phil




On Tue, 15 Jan 2019 at 14:39, RAJIV S  wrote:

Hi Team,


In DBCP 1.4 is there any property which can help to disconnect and
reconnect DB connection with in some specified time ?


Actually , We are using DBCP 1.4 version in our application. It's a
standalone Java processor code , developer some years ago, works well till
now.

Recently we encounter a problem that Oracle SWAM memory grows continuously
due to high usage . DBAs suggested to disconnect and reconnect db
connection from this java application with some specified time.

I am unable to find relevant property or  documentation help.

Please someone help me if anyone knows how do deal this situation.
--
Regards,

Rajiv.S






-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

[Phil Steitz]

On 9/9/18 12:16 AM, Mark Thomas wrote:

On 08/09/18 19:36, Mark Thomas wrote:




For the "- parking to wait for  <0x0006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
Though I see no other messages with that object in the thread dump.

Has anyone run into this? It seems like some sort of deadlock.

Do you still have the full thread dump? Can you post it somewhere (where
we can look at it)?

Thanks. Lots of threads waiting for the lock. None holding it. That is
very strange.

Some variation of this?

https://confluence.atlassian.com/jirakb/jira-applications-stall-due-to-stackoverflowerror-exception-941601100.html

Here is one thing to look at.  Note that the take request from 
borrowObject comes from

p = objectDeque.getIdleObjects().pollFirst();

This LinkedBlockingDeque method does not register the thread as 
waiting on the notEmpty condition in LinkedBlockingDeque the way 
that say

 p = objectDeque.getIdleObjects().pollFirst(
    borrowMaxWaitMillis, 
TimeUnit.MILLISECONDS);

does.

Phil


Mark

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org





-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Possible to get query strings from datasource?

[Phil Steitz]

On 5/4/18 5:15 PM, Shawn Heisey wrote:
> On 5/4/2018 4:49 PM, Martin Gainty wrote:
>> MG>have a chat with your DBA on BasicDataSource validationQuery
>> MG>if this was Oracle it would be something like select 1 from DUAL
> This has me chuckling.   Let me put on my DBA hat (which doesn't fit all
> that well)...
>
> The server is MySQL.  I set the validation query in my program.  It is
> "SELECT 1".  Here's everything I am setting on the BDS:
>
>   dsMain = new BasicDataSource();
>   // TODO: Put the driver name in the properties file.
>   dsMain.setDriverClassName("com.mysql.jdbc.Driver");
>   dsMain.setUrl(mainUrl);
>   dsMain.setUsername(dbUser);
>   dsMain.setPassword(dbPass);
>   dsMain.setValidationQuery(validationQuery);
>   dsMain.setValidationQueryTimeout(Const.FIVE_SECONDS / 1000);
>   dsMain.setMaxWaitMillis(Const.THIRTY_SECONDS);
>   dsMain.setMaxIdle(numShards);
>   dsMain.setMaxTotal(numShards * 5);
>   dsMain.setNumTestsPerEvictionRun(numShards * 5);
>   dsMain.setTimeBetweenEvictionRunsMillis(Const.THIRTY_SECONDS);
>   dsMain.setMinEvictableIdleTimeMillis(Const.ONE_MINUTE * 5);
>   dsMain.setTestOnCreate(true);
>   dsMain.setTestOnBorrow(true);
>   dsMain.setTestOnReturn(true);
>   dsMain.setTestWhileIdle(true);
>
> We don't have a DBA position in the company.  I'm the primary sysadmin,
> so the database servers are under my care, and one of the developers is
> designated as the gatekeeper for changes to the databases themselves, to
> keep things sane and not broken.
>
> Here's what I did.  Because all database usage is handled through one
> class, it was easy to find all the places where a query is executed.  I
> added this field to that class:
>
>     /** A map for tracking current SQL statements. */
>     private static final Map sqlTracker = Collections
>             .synchronizedMap(new HashMap());
>
> I really wanted to use the diamond operator here, as I have done when
> using synchronizedMap in other programs.  But eclipse complains about
> that unless I set the compiler to produce code for Java 8.  I have ONE
> system building/running this under Java 7, so I have to keep it like this.
>
> Then before every query execution (after the Connection and Statement
> were successfully created), I added this:
>
>   sqlTracker.put(conn, query);
>
> And in the helper method that always gets used for closing JDBC
> connections, I added this:
>
>   sqlTracker.remove(conn);
>
> This is the "getPoolStats()" method:
>
> https://apaste.info/v0UO
>
> The implementation looks clean to me, but I need something bad to happen
> before I will know if it's actually working.
>
> I could improve it by tracking queries separately for the main
> datasource and the master datasource, but just having the SQL queries,
> even without knowing which pool it comes from, is a nice addition.

Above looks like it will work, but it introduces a synchronization
bottleneck in your app.  If you have high concurrency and lots of
queries being submitted, threads are going to line up waiting for
the lock on the map.

>From a throughput / app impact perspective, I think you would be
better off using MySQL's show processlist, which you can get from
the driver (see SO link I posted).  The sync cost of maintaining
that is already being paid by the engine.  That will work as long as
you can identify the queries from you app by the user id. 

Phil
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Possible to get query strings from datasource?

[Phil Steitz]

On 5/4/18 12:16 PM, Shawn Heisey wrote:
> On 5/4/2018 10:38 AM, Phil Steitz wrote:
>> The only thing close to this that you can do now is if you have
>> prepared statement pooling enabled, then via the JMX instrumentation
>> of commons pool you can see the output of statement.toString().  
>> See listAllObjects in GenericObjectPoolMXBean.  Again, the pool you
>> need to walk to get this is the prepared statement pool associated
>> with an individual connection, not the connection pool.
> Is this an answer to both parts of my query?  Only the second part was
> concerned with prepared statements.  The first part was about getting
> currently executing SQL queries from BasicDataSource.

The answer to that is no, you can't get it directly from any BDS method.
>
> I am not using prepared statements at this time.  I will not be using
> them anytime soon.  The plan that includes prepared statements is a
> complete rewrite of the application from scratch, to eliminate some
> design problems.
>
> I was using PoolingDataSource in the past.  Because setting that up
> involves explicitly configuring the object pool, I did have access to
> listAllObjects, but since switching to BasicDataSource (which appears to
> be the recommended choice), that no longer seems to be available.

You can access it via JMX, but what you will see is just the
connection meta-data, what is exposed in PoolableConnectionMXBean.
>
> The "setAccessToUnderlyingConnectionAllowed" method implies that it
> should be possible to access the connections ... but I can't figure out
> how to do that.  Even if I manage that, I haven't yet found a way to
> descend from the connection to get the query.

If you cast the connection returned by BDS as a DelegatingConnection
and have setAccessToUnderlyingConnectionAllowed set to true, you can
use the getInnermostDelegate method to get the underlying physical
connection.  In general, you are not going to be able to get to
queries that way, though, as all you will have is the
javax.sql.Connection methods. 
>
> I acknowledge the possibility of an XY problem here, so here's what I'm
> trying to do:
>
> When my application fires off an alarm email, I want that email to
> include all SQL queries the application is executing at the moment of
> the alarm.
>
> Because all DB access is coordinated through a single class (which I
> creatively called "Database"), I can probably invent my own way of
> tracking this info, but if I can do it through dbcp, that's one less
> thing I have to write/debug.
I don't think DBCP is going to be much help there.  Another thing
you might consider is setting up a database user ID for the app and
using database utilities to dump the queries in progress for that
user when your alarm triggers.  Depending on the db / jdbc driver,
you may be able to do this from within your Java app.  See, e.g. [1]

Phil

[1]
https://stackoverflow.com/questions/14036690/dynamic-output-for-show-processlist-in-mysql-using-java/14036759#14036759

>   The alarm subroutines can only reach
> things that have access to the datasource, not anything below that.
>
> I think the way I would do it myself is using a Map<Connection, String>
> object built with Collections.synchronizedMap.  When I close a
> connection (return it to the pool), I can remove its entry from that
> tracking map.
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Possible to get query strings from datasource?

[Phil Steitz]

On 5/4/18 12:49 AM, Shawn Heisey wrote:
> Using dbcp 2.2.0 with pool 2.5.0.
>
> If the only object I have access to is BasicDataSource, is it
> possible to obtain the SQL queries being executed by active
> connections in the pool right now?  If so, how would I go about
> doing that?
>
> At the moment I'm not using PreparedStatement, but I have plans to
> do so in the future.  When that happens, I will be interested in
> finding the actual query being executed, not the query with the
> question mark placeholders.
>

The only thing close to this that you can do now is if you have
prepared statement pooling enabled, then via the JMX instrumentation
of commons pool you can see the output of statement.toString().  
See listAllObjects in GenericObjectPoolMXBean.  Again, the pool you
need to walk to get this is the prepared statement pool associated
with an individual connection, not the connection pool.

Phil
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] troubleshooting pool activity (tomcat version)

[Phil Steitz]

On 3/22/18 6:14 PM, Shawn Heisey wrote:
> First thing to do is thank you again for taking the time to help me. 
> Apache has great communities.
>
> On 3/22/2018 5:38 PM, Phil Steitz wrote:
>> You must be looking at documentation describing how to use the
>> alternative pool mentioned above (tomcat-jdbc).  The config you
>> posted is correct for DBCP.
> I'm looking at Tomcat documentation.
>
> https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html

That's the documentation for the alternative pool.  Use this instead:
https://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html#Database_Connection_Pool_(DBCP)_Configurations

>
> The tomcat is the one included with Liferay 6.2.  It is 7.0.42.

>From here
http://svn.apache.org/repos/asf/tomcat/tc7.0.x/tags/TOMCAT_7_0_42/build.properties.default

You can see the bundled DBCP / Pool are 1.4/1.5.7 which are the
latest compatible versions.
>
>> Don't look at DBCP 2 code for troubleshooting the code you are
>> running.  Either look at the repackaged sources inside the tomcat
>> source, or find the version in the tomcat build files and go to the
>> old DBCP / pool sources referenced there.
> I have figured that out.  Felt pretty dumb when I realized I wasn't
> looking at code from the correct Tomcat version.
>
>> Of course, you *should* upgrade both TC and the DBCP it ships so you
>> *can* look at that (much better) code.  See below for one reason why.
> I hear what you're saying, and don't disagree ... but this is not the
> kind of environment where I can just do an upgrade, even though
> upgrading might make it all better.
>
> We didn't download Tomcat -- we downloaded Liferay, which came with a
> specific version of Tomcat already included.  Upgrading any significant
> component (liferay, tomcat, and others) runs the risk that when we
> restart the service, our web application won't work any more.  For any
> upgrade, we have to spend a lot of resources trying the upgrade in a
> staging environment, so we can be sure that everything still works. 
> Because that's very time-consuming, we tend to not do a lot of
> upgrading, at least of significant components, and our versions get
> REALLY old.
>
> This is also why I'm hesitant to move away from Tomcat's DBCP
> implementation to Commons DBCP (particularly version 2), even though
> that's exactly what I want to do.  Switching to a different library
> might work seamlessly ... or it might completely break the application. 
> Our customers get REALLY irritated when the websites we've built for
> them don't work!

Using DBCP2 with TC 7 won't work.  You need TC 8+ for that.
>
>> One thing that could be going on is that in the old 1.x DBCP, 
>> abandoned connection removal only happens when borrows are
>> attempted.  So if you check out a lot of connections, abandon them
>> and don't ask for more, they won't get closed as abandoned until you
>> borrow a new one.  In DBCP 2, the removeAbandoned property is split
>> into two different properties:  removeAbandonedOnBorrow (the old
>> behavior) and removeAbandonedOnMaintenance.  The second one makes
>> abandoned connection removal run on pool maintenance (so will not
>> have to wait until a borrow is attempted).
> I don't know if anyone needs me to actually back up and describe what's
> happening that led me down this rabbit hole, but that's what I'm going
> to do:
>
> The master MySQL server in our environment has a max connection limit
> configured at 600 connections.
>
> Every now and then, we start getting website failures, because all the
> connections are in use and the connection pools can't make any more
> connections.  Looking at the connections on the MySQL side, the vast
> majority are idle (the command is "Sleep" on the server processlist),
> and have been idle for several hours.
>
> There are five main webservers and a handful of ancillary systems that
> also connect to the database.  When the problem happens, the connection
> count from each webserver has gotten up near 100, and sometimes over
> 100.  The surplus of connections are definitely the ones configured in
> Tomcat.  Liferay has its own DB config for its own data (using c3p0 for
> pooling), and although I often see a higher number of connections to
> that database than I would excpect, I've never seen the idle time on
> those connections above one minute, so I'm not concerned about that
> pool, beyond some minor tweaks.
>
> The frequency of the connection-related failures has been increasing, so
> in response, I have set up monitoring that will send us an alarm when
> the server reaches 550 connections.  This has allowed us to kill idle
> connections and prevent customer-visible pr

Re: [DBCP] troubleshooting pool activity (tomcat version)

[Phil Steitz]

On 3/21/18 12:15 PM, Shawn Heisey wrote:
> On 3/21/2018 1:31 AM, Mark Thomas wrote:
>>> and that we need to
>>> change the factory on our pool definitions.
>> I believe not.
> Tomcat's documentation seems to disagree with this point. It
> specifically says to use org.apache.tomcat.jdbc.pool.DataSourceFactory,
> and doesn't list any other valid choices.

You must be looking at documentation describing how to use the
alternative pool mentioned above (tomcat-jdbc).  The config you
posted is correct for DBCP.

> But we have
> org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory in our
> configuration.  I've found questions from people whose Tomcat
> installations didn't even contain that class, where the answers said to
> switch to the factory in the documentation.  Apparently deb-based
> packages of Tomcat do not contain the latter class.
>
> I found a historical document for 7.0.42 with Google and it also shows
> that the factory should be set to the DataSourceFactory, not the
> BasicDataSourceFactory that we have.
>
> I do see "removeAbandoned" in the property constants in the 7.0 source
> code for BasicDataSourceFactory  I had thought that it wasn't there,
> but I realized that I was looking at trunk code when I came to that
> conclusion, and that BasicDataSourceFactory is in a dbcp2 package in
> trunk.  
Don't look at DBCP 2 code for troubleshooting the code you are
running.  Either look at the repackaged sources inside the tomcat
source, or find the version in the tomcat build files and go to the
old DBCP / pool sources referenced there.

Of course, you *should* upgrade both TC and the DBCP it ships so you
*can* look at that (much better) code.  See below for one reason why.
> So I'm not sure what to think, but I can say that the abandoned
> connection handling does not appear to actually be working.  So either
> my configuration is wrong, or the factory that we are using is ignoring
> part of the config.

One thing that could be going on is that in the old 1.x DBCP, 
abandoned connection removal only happens when borrows are
attempted.  So if you check out a lot of connections, abandon them
and don't ask for more, they won't get closed as abandoned until you
borrow a new one.  In DBCP 2, the removeAbandoned property is split
into two different properties:  removeAbandonedOnBorrow (the old
behavior) and removeAbandonedOnMaintenance.  The second one makes
abandoned connection removal run on pool maintenance (so will not
have to wait until a borrow is attempted).

Phil
>
> Thanks,
> Shawn
>
> p.s. I sent an earlier draft of this message, but did so from the wrong
> email address.  If the moderators decide to allow that message through,
> then you may see an almost-duplicate of this message.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] troubleshooting pool activity (tomcat version)

[Phil Steitz]

On 3/20/18 7:21 PM, Shawn Heisey wrote:
> I've written before, trying to track down problems with our
> database server getting too many connections.
>
> Based on what I can see of how my programs (using dbcp2) are
> working, everything seems to be fine there.  I've added logging to
> tell me how many idle and active connections there are in the
> pool, and those numbers stay low.
>
> So now I need to track down what's happening in our webapps, most
> of which are Liferay-based, and all of which are running in
> Tomcat.  And I've learned that they're using the Tomcat
> implementation of dbcp for database access.  I think it's Tomcat
> version 7, but I will need to check to make sure.
>
> I've figured out how to log the number of active and idle
> connections, by casting the DataSource to the tomcat dbcp object
> actually in use.  Once I can get the developers to actually update
> a production system with that code, I will be able to see whether
> (as I suspect) the "active" count in the pool is staying
> abnormally high.  I'm betting that there's somewhere in the webapp
> code that a connection is retrieved from the pool, used for some
> work, and never closed.
>
> I have a main question, and then a semi-related but very different
> question.
>
> Main Question:  Does dbcp by chance record a stacktrace of the
> code that requests a connection from the pool?  I would like to
> poke my way through the active connections (entry point being the
> DataSource implementation), and ask them where in our code they
> were requested.  I have to do this in the Tomcat fork of dbcp, and
> I know I'm not on a Tomcat mailing list, but I'm hoping that
> whatever you can tell me will apply to that version too.
First, find out what version of tomcat you are running.  Then look
in the tomcat build file sources for the properties that define the
dbcp and pool versions being used.  In TC 7, I am pretty sure the
repackaged sources were always from release tags.  Ask again if you
have trouble locating the dbcp/pool versions once you know the TC
version.

In DBCP 1.x (what TC 7 used), abandoned connection tracking was in
the AbandonedObjectPool bundled with DBCP.  Tracking can be turned
on by configuration of BasicDataSource to remove abandoned
connections and to log them.  When connections are considered
abandoned and closed, the stack trace of the code that created them
is logged.  That requires that they actually go past the abandoned
connection timeout, though and get closed by the pool.

> If I can get a stacktrace of where each connection was requested,
> I can pinpoint problematic code a lot faster.  There is a LOT of
> code that uses the database, scattered across a lot of git
> repositories.  If I could grep the code easily to find them all, I
> would.
>
> Side question: Why has Tomcat maintained what looks like a fork of
> dbcp for such a long time period?  If they really believe their
> implementation has an advantage, it seems like everyone would
> benefit if they worked to get the upstream library to offer the
> same advantages.  Am I drifting into flamewar territory by even
> wondering about this?  I did find the tomcat documentation page
> about their connection pool.  It basically reads to me as "Commons
> DBCP is substandard and bloated.  These are the many ways that our
> implementation is better."

There are two different things going on here.  The first is the
repackaged DBCP sources that tomcat ships as the default connection
pool.  That is not really a fork - just repackaging.  In the early
days of TC8, DBCP was not releasing fast enough, so IIRC a few TC
releases shipped pre-release DBCP sources, but there was never
really any forking going on.  The second thing is the tomcat's
"jdbc-pool" module that is an alternative to DBCP.  The content on
the web page you reference is obsolete and probably should be
fixed.  It refers back to 1.x versions of DBCP that used 1.x
versions of pool.  Those were in fact slow and buggy.  The 2.x
versions of DBCP have comparable performance (slightly slower, but
not much) to jdbc-pool (mostly because pool 2.x is much faster).  
Depending on which version of TC 7 you are running, jdbc-pool for
that version might be materially faster than DBCP version shipped
with it.

Phil
>
> https://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html
>
> I haven't compared DBCP with other pool implementations, but my
> work with DBCP has never run into any problems that weren't my own
> fault.  This mailing list has shown a high degree of patience with
> my dumb questions.
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [Math] How to do Polynomial Regression ?

[Phil Steitz]

On 3/12/18 10:15 AM, Debraj Manna wrote:
> Crossposting from stackoverflow
> 
> as I did not get any reply there
>
> Can someone let me know how I can do Polynomial Regression with Apache Maths
>  3.6.1
> ?
>
> Below are the data points I used for my testing
>
> 60735214881.391304  152025480.0060697824142.469570
> 152025840.0060651182200.208694
> 152026200.0060684367132.939130
> 152026560.0060676588613.008700
> 152026920.0060641816564.869570
> 152027280.0060604714824.233510
> 152027640.0060580042814.330440
> 152028000.0060536134542.469570
> 152028360.0060566323732.034780
> 152028720.0060578775249.252174
> 152029080.0060547382844.104350
> 152029440.0060536776546.802160
> 152029800.0060474342718.330440
> 152030160.0060452725477.286960
> 152030520.0060486821569.669560
> 152030880.0060247997139.995674
> 152031240.0060248432181.426090
> 152031600.0060217476247.373920
> 152031960.0060170744493.634780  152032320.00
>
> My code looks like below
>
> private void polynomialFitter(List pointlist) {
> final PolynomialCurveFitter fitter = PolynomialCurveFitter.create(2);
> final WeightedObservedPoints obs = new WeightedObservedPoints();
> for (List point : pointlist) {
> obs.add(point.get(1), point.get(0));
> }
> double[] fit = fitter.fit(obs.toList());
> System.out.printf("\nCoefficient %f, %f, %f", fit[0], fit[1], fit[2]);
> }
>
> The coefficients are reported as
>
> Coefficient 12.910025, 0.00, 0.00
>
> But these does not seem to be quite correct. If I use the same dataset
> in Online
> Polynimal Regression  and
> in archanoid online regression  - both
> reports same value as 654623237474.68250993904929103762,
> 28.75921919628759991574, -0.023885199278
>
> Can someone let me know what is going wrong? I have seen this question
> 
>  but that is not helping me?

Polynomial regression is not the same as curve fitting.  To do
polynomial regression in Commons Math, use the
OLSMultipleLinearRegression class, using, X, X^2 etc as the
independent variables (as your second reference above shows).

Phil




>
> Thanks,
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Connection pool not behaving as I expect

[Phil Steitz]

> On Mar 1, 2018, at 3:33 PM, Shawn Heisey  wrote:
> 
> We have been having some problems lately where our MySQL server hits the
> max connection limit (600) and then everything breaks.  When I look into
> the problem, I find that our application servers have each made nearly a
> hundred connections to the DB and haven't closed any of them for hours.
> 
> I'm also using connection pooling in my programs, with the latest DBCP
> version.  Those servers don't open nearly as many connections, and have
> idle eviction to keep the connection count down.  But when the limit is
> reached, these programs suddenly stop working too.
> 
> Investigating these problems, I manage to get connected and kill off the
> surplus of idle connections, and everything starts working.
> 
> Today, a couple of days after the last incident, I realized that we
> should *NOT* be having these problems -- because we're using connection
> pooling.  The application has open and idle connections to the DB server
> ... so why is trying to open MORE connections (and obviously failing)
> instead of using one of the perfectly good connections that's already
> sitting there, unused?
> 
> I'm writing here specifically for DBCP on my programs, so I know you
> guys probably can't help with Tomcat's connection pooling ... but for
> either case my question stands:  Why isn't connection pooling doing its job?

Best to start by posting your pool config.  One thing to bear in mind is that 
“idle” from The dB perspective does not mean the same thing as idle from the 
pool’s perspective.  In addition to genuinely abandoned connections, another 
way that applications can hog dB connections is to check them out and hold them 
for a long time while not using them.  On the dB engine side these will show as 
idle, but as they are checked out to clients they are not available in the 
pool.  When you are having the problem, what does dB one report as numidle?

Phil
> 
> Thanks,
> Shawn
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 1/9/18 5:09 PM, Shawn Heisey wrote:
> On 1/9/2018 11:50 AM, Phil Steitz wrote:
>> thread 1: checkOpen - sees true
>> thread 2: close the DelegatingConnection  (there is no sync to
>> prevent this)
>> thread1 : createStatement - bang!
>> thread1 : isClosed() returns true
>>
>> DBCP is not really safe to use that way - i.e., really the intended
>> setup is that individual connection handles are not concurrently
>> accessed by multiple threads.  Is it possible something like this is
>> going on?  Note that what I am talking about here is two different
>> threads holding references to the same connection handle - i.e., no
>> trips back through the pool.
> I am about 99 percent sure that I never pass Connection objects between
> threads.  What code I've looked at borrows, uses, and closes the
> Connection with a single thread.  I have some helper methods that take a
> connection, but in all the code I've looked at so far, it's all running
> in the same thread that borrowed the Connection in the first place.
>
> I can tell you for sure that my *intent* when I wrote the code was to
> always handle a Connection lifecycle in one single thread.  I wasn't
> absolutely sure that sharing the object between threads would cause
> problems, but doing so seemed like a bad idea.

The other possibility here is a pool bug causing the same connection
to be handed out to two different clients.  The DefaultPooledObject
machinery and other guards in [pool] make that unlikely.  There are
tests in [pool] that verify that this does not happen. 
Specifically, the unit test named testNoInstanceOverlap is set up to
detect "instance sharing."  I modified it to have a setup similar to
yours (lots of concurrent eviction, maxIdle < maxTotal, etc.) and
soaked it with a range of different configs and could not get
"overlap" to happen.  Feel free to play with that test and see if
you can get a failure.  I can't see how this can happen, honestly;
thoug.  If your code can't be sharing across threads, I don't see
how the combination of the stack trace posted and isClosed returning
true immediately after could happen.

Phil
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 1/9/18 1:56 PM, Phil Steitz wrote:
> On 1/9/18 11:50 AM, Phil Steitz wrote:
>> On 1/8/18 4:23 PM, Shawn Heisey wrote:
>>> On 11/22/2017 5:00 PM, Phil Steitz wrote:
>>>> If the problem is the evictor closing a connection and having that
>>>> connection delivered to a client, the problem is almost certainly in
>>>> pool.  The thread-safety of the pool in this regard is engineered in
>>>> DefaultPooledObject, which is the wrapper that pool manages and
>>>> delivers to DBCP.  When the evictor visits a PooledObject (in
>>>> GenericObjectPool#evict) it tries to start the eviction test on the
>>>> object by calling its startEvictionTest method.  This method is
>>>> synchronized on the DefaultPooledObject.  Look at the code in that
>>>> method.  It checks to make sure that the object is in fact idle in
>>>> the pool.  The other half of the protection here is in
>>>> GenericObjectPool#borrowObject, which is what PoolingDataSource
>>>> calls to get a connection.  That method tries to get a PooledObject
>>>> from the pool and before handing it out (or validating it), it calls
>>>> the PooledObject's allocate method.  Look at the code for that in
>>>> DefaultPooledObject.  That method (also synchronized on the
>>>> PooledObject) checks that the object is not under eviction and sets
>>>> its state to allocated.  That is the core sync protection that
>>>> *should* make it impossible for the evictor to do anything to an
>>>> object that has been handed out to a client.
>>> I see the synchronization you're talking about here.  It appears that
>>> all of the critical methods in DefaultPooledObject are synchronized (on
>>> the object).
>>>
>>> If you're absolutely certain that DefaultPooledObject is involved with
>>> all of the implementation my code is using, it all looks pretty complete
>>> to me. 
>> Yes, the code you posted at the top of the thread uses a
>> PoolableConnectionFactory as the object factory for the pool.  You
>> can see that PCF's makeObject returns a DefaultPooledObject, so that
>> much is certain.
>>>  So I'm really curious as to why the connection is getting
>>> closed.  I have seen the problem only minutes after restarting my
>>> program, so it seems unlikely that the server side is closing the
>>> connection, since the timeout for that is 8 hours.
>> I looked back at the initial stack trace and I noticed something
>> that I had not noticed before.
>>
>> This line
>>
>> org.apache.commons.dbcp2.DelegatingConnection.createStatement(DelegatingConnection.java:262)
>>
>> means that checkOpen() succeeded.  That, combined with your
>> statement above that isClosed() returns true on a failed connection
>> means that there might be concurrent access to the
>> DelegatingConnections happening.  It looks like the sequence might
>> have been:
>>
>> thread 1: checkOpen - sees true
>> thread 2: close the DelegatingConnection  (there is no sync to
>> prevent this)
>> thread1 : createStatement - bang!
>> thread1 : isClosed() returns true
>>
>> DBCP is not really safe to use that way - i.e., really the intended
>> setup is that individual connection handles are not concurrently
>> accessed by multiple threads.  Is it possible something like this is
>> going on?  Note that what I am talking about here is two different
>> threads holding references to the same connection handle - i.e., no
>> trips back through the pool.
> I just noticed another thing in [pool] that might have something to
> do with this.  It's probably best to investigate what I have in mind
> on the dev list.  I will post a summary / ticket reference here if
> it turns out I this is a bug.

Sorry for the noise.  Bug idea evaporated when I dug into it.

Phil

>
> Phil
>> Phil
>>> I did add the code a while back to test on create, borrow, return, and
>>> while idle, but it turns out that I hadn't actually pulled it down to
>>> the test server and recompiled.  That is now done, so we'll see if that
>>> makes any difference.
>>>
>>> If testing the connection on pool actions does make a difference, then
>>> what is your speculation about what was happening when I ran into the
>>> closed connection only minutes after restart, and would it be worthy of
>>> an issue in Jira?  The only theory I had was a race condition between
>>> eviction and borrowing, but unless there's something amiss in how all
>>> the object inheritance 

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 1/9/18 11:50 AM, Phil Steitz wrote:
> On 1/8/18 4:23 PM, Shawn Heisey wrote:
>> On 11/22/2017 5:00 PM, Phil Steitz wrote:
>>> If the problem is the evictor closing a connection and having that
>>> connection delivered to a client, the problem is almost certainly in
>>> pool.  The thread-safety of the pool in this regard is engineered in
>>> DefaultPooledObject, which is the wrapper that pool manages and
>>> delivers to DBCP.  When the evictor visits a PooledObject (in
>>> GenericObjectPool#evict) it tries to start the eviction test on the
>>> object by calling its startEvictionTest method.  This method is
>>> synchronized on the DefaultPooledObject.  Look at the code in that
>>> method.  It checks to make sure that the object is in fact idle in
>>> the pool.  The other half of the protection here is in
>>> GenericObjectPool#borrowObject, which is what PoolingDataSource
>>> calls to get a connection.  That method tries to get a PooledObject
>>> from the pool and before handing it out (or validating it), it calls
>>> the PooledObject's allocate method.  Look at the code for that in
>>> DefaultPooledObject.  That method (also synchronized on the
>>> PooledObject) checks that the object is not under eviction and sets
>>> its state to allocated.  That is the core sync protection that
>>> *should* make it impossible for the evictor to do anything to an
>>> object that has been handed out to a client.
>> I see the synchronization you're talking about here.  It appears that
>> all of the critical methods in DefaultPooledObject are synchronized (on
>> the object).
>>
>> If you're absolutely certain that DefaultPooledObject is involved with
>> all of the implementation my code is using, it all looks pretty complete
>> to me. 
> Yes, the code you posted at the top of the thread uses a
> PoolableConnectionFactory as the object factory for the pool.  You
> can see that PCF's makeObject returns a DefaultPooledObject, so that
> much is certain.
>>  So I'm really curious as to why the connection is getting
>> closed.  I have seen the problem only minutes after restarting my
>> program, so it seems unlikely that the server side is closing the
>> connection, since the timeout for that is 8 hours.
> I looked back at the initial stack trace and I noticed something
> that I had not noticed before.
>
> This line
>
> org.apache.commons.dbcp2.DelegatingConnection.createStatement(DelegatingConnection.java:262)
>
> means that checkOpen() succeeded.  That, combined with your
> statement above that isClosed() returns true on a failed connection
> means that there might be concurrent access to the
> DelegatingConnections happening.  It looks like the sequence might
> have been:
>
> thread 1: checkOpen - sees true
> thread 2: close the DelegatingConnection  (there is no sync to
> prevent this)
> thread1 : createStatement - bang!
> thread1 : isClosed() returns true
>
> DBCP is not really safe to use that way - i.e., really the intended
> setup is that individual connection handles are not concurrently
> accessed by multiple threads.  Is it possible something like this is
> going on?  Note that what I am talking about here is two different
> threads holding references to the same connection handle - i.e., no
> trips back through the pool.

I just noticed another thing in [pool] that might have something to
do with this.  It's probably best to investigate what I have in mind
on the dev list.  I will post a summary / ticket reference here if
it turns out I this is a bug.

Phil
>
> Phil
>> I did add the code a while back to test on create, borrow, return, and
>> while idle, but it turns out that I hadn't actually pulled it down to
>> the test server and recompiled.  That is now done, so we'll see if that
>> makes any difference.
>>
>> If testing the connection on pool actions does make a difference, then
>> what is your speculation about what was happening when I ran into the
>> closed connection only minutes after restart, and would it be worthy of
>> an issue in Jira?  The only theory I had was a race condition between
>> eviction and borrowing, but unless there's something amiss in how all
>> the object inheritance works out, it looks like that's probably not it. 
>> Some kind of issue with the TCP stack in Linux (either on the machines
>> running my code or the MySQL server) is the only other idea I can think
>> of.  Or maybe a hardware/firmware issue, since it's likely that at least
>> one of the NICs involved is doing TCP offload.  I think that virtually
>> every NIC in our infrastructure has that featur

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 1/8/18 4:23 PM, Shawn Heisey wrote:
> On 11/22/2017 5:00 PM, Phil Steitz wrote:
>> If the problem is the evictor closing a connection and having that
>> connection delivered to a client, the problem is almost certainly in
>> pool.  The thread-safety of the pool in this regard is engineered in
>> DefaultPooledObject, which is the wrapper that pool manages and
>> delivers to DBCP.  When the evictor visits a PooledObject (in
>> GenericObjectPool#evict) it tries to start the eviction test on the
>> object by calling its startEvictionTest method.  This method is
>> synchronized on the DefaultPooledObject.  Look at the code in that
>> method.  It checks to make sure that the object is in fact idle in
>> the pool.  The other half of the protection here is in
>> GenericObjectPool#borrowObject, which is what PoolingDataSource
>> calls to get a connection.  That method tries to get a PooledObject
>> from the pool and before handing it out (or validating it), it calls
>> the PooledObject's allocate method.  Look at the code for that in
>> DefaultPooledObject.  That method (also synchronized on the
>> PooledObject) checks that the object is not under eviction and sets
>> its state to allocated.  That is the core sync protection that
>> *should* make it impossible for the evictor to do anything to an
>> object that has been handed out to a client.
> I see the synchronization you're talking about here.  It appears that
> all of the critical methods in DefaultPooledObject are synchronized (on
> the object).
>
> If you're absolutely certain that DefaultPooledObject is involved with
> all of the implementation my code is using, it all looks pretty complete
> to me. 

Yes, the code you posted at the top of the thread uses a
PoolableConnectionFactory as the object factory for the pool.  You
can see that PCF's makeObject returns a DefaultPooledObject, so that
much is certain.
>  So I'm really curious as to why the connection is getting
> closed.  I have seen the problem only minutes after restarting my
> program, so it seems unlikely that the server side is closing the
> connection, since the timeout for that is 8 hours.

I looked back at the initial stack trace and I noticed something
that I had not noticed before.

This line

org.apache.commons.dbcp2.DelegatingConnection.createStatement(DelegatingConnection.java:262)

means that checkOpen() succeeded.  That, combined with your
statement above that isClosed() returns true on a failed connection
means that there might be concurrent access to the
DelegatingConnections happening.  It looks like the sequence might
have been:

thread 1: checkOpen - sees true
thread 2: close the DelegatingConnection  (there is no sync to
prevent this)
thread1 : createStatement - bang!
thread1 : isClosed() returns true

DBCP is not really safe to use that way - i.e., really the intended
setup is that individual connection handles are not concurrently
accessed by multiple threads.  Is it possible something like this is
going on?  Note that what I am talking about here is two different
threads holding references to the same connection handle - i.e., no
trips back through the pool.

Phil
>
> I did add the code a while back to test on create, borrow, return, and
> while idle, but it turns out that I hadn't actually pulled it down to
> the test server and recompiled.  That is now done, so we'll see if that
> makes any difference.
>
> If testing the connection on pool actions does make a difference, then
> what is your speculation about what was happening when I ran into the
> closed connection only minutes after restart, and would it be worthy of
> an issue in Jira?  The only theory I had was a race condition between
> eviction and borrowing, but unless there's something amiss in how all
> the object inheritance works out, it looks like that's probably not it. 
> Some kind of issue with the TCP stack in Linux (either on the machines
> running my code or the MySQL server) is the only other idea I can think
> of.  Or maybe a hardware/firmware issue, since it's likely that at least
> one of the NICs involved is doing TCP offload.  I think that virtually
> every NIC in our infrastructure has that feature and that Linux enables it.


>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 11/22/17 2:43 PM, Shawn Heisey wrote:
> On 11/22/2017 12:04 PM, Phil Steitz wrote:
>> On 11/22/17 9:43 AM, Shawn Heisey wrote:
>>> I do have results from the isClosed method when the problem happens. 
>>> That method *does* return true. 
>> That points to a Pool or DBCP bug, assuming you are sure that no
>> other thread has a reference to the PoolableConnection or some other
>> code path did not call close on it before you tested isClosed.  If
>> you are sure this is not happening, you should open a DBCP JIRA
>> (which may end up reassigned to pool).  Ideal would be to have a
>> test case that makes it happen.
> I would absolutely love to have a test case that can reproduce it, but
> since I haven't got any idea what the root of the problem is, I wouldn't
> know how to write such a test case.
>
> What I'd really like to do is be able to look over dbcp2 and pool2 code
> to see if I can spot a problem, but I'm having a hard time following the
> code.  I expected to find some kind of synchronization in the code
> branching from getConnection() to prevent different threads from being
> able to step on each other, but haven't seen any so far.  I can't tell
> if this means I'm looking in the wrong place or not.  The object
> inheritance is pretty extensive, so I'm having a hard time finding the
> right place to look.
>
> If it turns out that there is zero synchronization happening between the
> idle eviction thread and the depths of the code for things like
> getConnection, then I don't see how any kind of guarantee can be made. 
> So far the synchronization object in the eviction thread only seems to
> pair with other parts of eviction, NOT with anything else in the library.
>
> If the testXXX flags I've enabled do eliminate the problems I'm seeing,
> that's awesome, and it's *A* solution, but I think I'm still running
> into some kind of issue that needs to be fixed.  I just need to figure
> out whether it's dbcp2, pool2, or something in my own environment.  I'm
> willing to entertain the idea that it's my environment, but based on
> everything that I understand about my own code and our database servers
> (and I fully admit it's circumstantial evidence), it points to a problem
> with the idle eviction thread.
>
> I believe you when you say that the *intent* is for idle eviction to
> never close/evict a connection that's been requested from the pool.  I
> would like to verify whether the intent and what's actually implemented
> are the same.  If they're not the same, then I would like to attempt a
> patch.  I'm going to need help in figuring out exactly where I should be
> looking in the code for dbcp2 and pool2.

If the problem is the evictor closing a connection and having that
connection delivered to a client, the problem is almost certainly in
pool.  The thread-safety of the pool in this regard is engineered in
DefaultPooledObject, which is the wrapper that pool manages and
delivers to DBCP.  When the evictor visits a PooledObject (in
GenericObjectPool#evict) it tries to start the eviction test on the
object by calling its startEvictionTest method.  This method is
synchronized on the DefaultPooledObject.  Look at the code in that
method.  It checks to make sure that the object is in fact idle in
the pool.  The other half of the protection here is in
GenericObjectPool#borrowObject, which is what PoolingDataSource
calls to get a connection.  That method tries to get a PooledObject
from the pool and before handing it out (or validating it), it calls
the PooledObject's allocate method.  Look at the code for that in
DefaultPooledObject.  That method (also synchronized on the
PooledObject) checks that the object is not under eviction and sets
its state to allocated.  That is the core sync protection that
*should* make it impossible for the evictor to do anything to an
object that has been handed out to a client.

The logical place to start to get a test case that shows this
protection failing is to just set up a pool with very aggressive
eviction config (very small idle object timeout), frequent eviction
runs and a lot of concurrent borrowing.  Make sure the factory's
destroy method does something to simulate what PCF does to mark the
object as dead and see if you get any corpses handed out to
borrowers.  Also make sure that there are enough idle instances in
the pool for the evictor to visit.  For that, you probably want to
vary the borrowing load.  You can set up jmx to observe the pool
stats to see how many are idle at a given time or just log it using
the getNumIdle.  A quick look at the existing pool2 test cases does
not show exactly that scenario covered, so it would be good to add
in any case.

Phil


>
> Thanks,
> Shawn
>
>
> 

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 11/22/17 9:43 AM, Shawn Heisey wrote:
> On 11/21/2017 1:01 PM, Phil Steitz wrote:
>> As I said in my first response, the most common explanation for this
>> kind of exception when using DBCP is that the underlying physical
>> connection is closed on the server (or network) side without DBCP
>> knowing about it.  And the most common cause of that is server-side
>> idle connection timeout.
> I do have results from the isClosed method when the problem happens. 
> That method *does* return true.
>
> I am pretty sure that the server-side idle timeout is at MySQL's default
> of 28800 seconds, or 8 hours.  The problem I am experiencing has
> happened only a few minutes after starting my program, so in that case,
> I would not expect a server-side timeout to happen.
>
>> It just occurred to me that since you do not have any of the testXxx
>> flags set to true, DBCP is never actually exercising the
>> connections.
> I have now enabled the flags for testing on create, borrow, return, and
> while idle.  If this does eliminate problems, do you think my theory
> might be correct, or that it's probably something else?  The
> testWhileIdle flag does seem a little excessive, but I figured I would
> go ahead with it.  The "SELECT 1" validation query should always be fast
> as long as everything's working right ... and if it's not working right,
> I am not going to mind a few seconds passing before the problem is reported.
>
>> Is it possible that sometimes your code checks out a
>> connection from the pool but does not use it?
> I'm only calling the datasource getConnection in a getConnection method
> of my own, and that method is only used in the methods which perform
> work on a connection.

You're not storing the connection somewhere in a possibly not
thread-safe way, are you? 

Phil
>   It seems unlikely that I am checking out a
> connection and never using it, but to be certain I would need to do a
> full review of the code in my Database class.  I should probably do that
> anyway, but it's a fair amount of work, so I've been avoiding it.
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 11/22/17 9:43 AM, Shawn Heisey wrote:
> On 11/21/2017 1:01 PM, Phil Steitz wrote:
>> As I said in my first response, the most common explanation for this
>> kind of exception when using DBCP is that the underlying physical
>> connection is closed on the server (or network) side without DBCP
>> knowing about it.  And the most common cause of that is server-side
>> idle connection timeout.
> I do have results from the isClosed method when the problem happens. 
> That method *does* return true.

That points to a Pool or DBCP bug, assuming you are sure that no
other thread has a reference to the PoolableConnection or some other
code path did not call close on it before you tested isClosed.  If
you are sure this is not happening, you should open a DBCP JIRA
(which may end up reassigned to pool).  Ideal would be to have a
test case that makes it happen.
>
> I am pretty sure that the server-side idle timeout is at MySQL's default
> of 28800 seconds, or 8 hours.  The problem I am experiencing has
> happened only a few minutes after starting my program, so in that case,
> I would not expect a server-side timeout to happen.

Makes sense, so can rule that out.
>
>> It just occurred to me that since you do not have any of the testXxx
>> flags set to true, DBCP is never actually exercising the
>> connections.
> I have now enabled the flags for testing on create, borrow, return, and
> while idle.  If this does eliminate problems, do you think my theory
> might be correct, or that it's probably something else? 

Your theory that it is a pool bug?  If adding those tests makes the
problem go away, that might just be due to timing.

>  The
> testWhileIdle flag does seem a little excessive, but I figured I would
> go ahead with it.  The "SELECT 1" validation query should always be fast
> as long as everything's working right ... and if it's not working right,
> I am not going to mind a few seconds passing before the problem is reported.
>
>> Is it possible that sometimes your code checks out a
>> connection from the pool but does not use it?
> I'm only calling the datasource getConnection in a getConnection method
> of my own, and that method is only used in the methods which perform
> work on a connection.  It seems unlikely that I am checking out a
> connection and never using it, but to be certain I would need to do a
> full review of the code in my Database class.  I should probably do that
> anyway, but it's a fair amount of work, so I've been avoiding it.

Code review is always great ;) but given that the problem happens
close to startup, it is not likely that the connections are being
closed server-side.  The fact that isClosed is returning true when
the problem happens also rules that out - it means that DBCP knows
that the connection is closed. That doesn't happen when server-side
timeouts happen.

>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 11/21/17 10:58 AM, Shawn Heisey wrote:
> On 11/17/2017 3:27 PM, Phil Steitz wrote:
>> OK, sorry I misread your initial post.  That means either the pool
>> agrees that the connection is closed (which would point to a pool or
>> DBCP bug) or the driver's isValid() on the underlying physical
>> connection is returning true (without throwing).  If it's not too
>> hard to do, it would be good to also log what comes back from
>> isClosed() on the DelegatingConnection (what dsMaster returns).  If
>> that returns true, then either this is a DBCP or pool bug or you
>> have somehow closed the DelegatingConnection via another reference
>> or something.  If it returns false, that means the pool thinks the
>> connection is open so it is unlikely that the pool closed it.
> I'm pretty sure (will have to check) that when my code uses the
> connections it gets from the pool, it does do a close on it when
> complete, but it's my understanding that this is how to return
> connections to the pool, that it doesn't *truly* close the connection as
> it would if I were using JDBC directly instead of via DBCP.

That is correct.
>
> I have changed my program to check isClosed instead of making the
> "SELECT 1" query on the invalid connection, and to log something if that
> returns true.  So far the problem hasn't happened again, even though I
> did set the eviction interval back to exactly one minute.

That will tell if something funny is going on.  Having isClosed
return true on a connection that you have just gotten from the pool
(and no other thread has messed with) would be DBCP/Pool badness.
>
>> What you have below is fine.  I just wanted to see the pool
>> configuration settings.  Looks OK, but might end up creating a lot
>> of connection churn having maxIdle set to 6 with maxTotal at 30.  I
>> guess the idea is that when it runs, you want to have an idle
>> connection available for each shard with the ability to add more up
>> to a total of 30, but you don't want those idle connections to stay
>> in the pool when the work isn't being done.  Do you know how many
>> connections actually get opened during the work period?  If that is
>> more than a few more than 6, you are kind of defeating the purpose
>> of the pool by closing them all when they get returned (as the
>> maxIdle setting will force).
> Here's a screenshot showing the open connections on the DB server at a
> particular moment.  The connections in this case are coming from the
> server named "idxa1".  There are two copies of the program running --
> one where numShards is 2, and one where numShards is 6.  As you can see
> in the screenshot, there are twelve connections from idxa1.
>
> https://www.dropbox.com/s/g9swe4hrq19jbcc/mysql-dbcp-connections-workbench.png?dl=0
>
> As the program does its work, it uses one connection to the main server
> to gather some information and then it can open up numShards+1
> connections to the main server to actually handle updating the Solr
> index from the database.  (numShards refers to the number of cold
> shards, but there is one more shard in each index -- the hot shard).  On
> each cycle, position information is written to the master server.
>
> Because we had to do some work on a DB server a while back, the main
> pool and the master pool are both set to the same host -- the master
> server.  This is slightly unusual, but that configuration pre-dates the
> recent problems with closed connections by quite a bit of time. 
> Eventually I will be switching it back so the main server is one of the
> slaves, but I would like to figure this problem out first.
>
>> The fact that changing the eviction interval to not exactly coincide
>> with when the work happens makes the problem go away is troubling. 
>> I can see how that might improve performance, but it does make the
>> pool bug scenario more likely.
> Do you think I should be doing things differently than I currently do? 
> If so, please feel free to make recommendations.  I'm not really sure
> what your statement about performance is saying.  I'm not doing eviction
> for performance reasons -- in fact, it might actually *reduce*
> performance, because it's more likely that "getConnection" is going to
> have to actually spin up a new TCP connection to the database.  The time
> required for that doesn't worry me.

The performance comment was just to point out that if you run the
evictor exactly at the time when the work happens, there is some
contention between the evictor and the borrowing threads.  But this
has really been optimized in pool2, so you probably can't measure
it.  So basically forget that comment.  The fact that the problem
goes away when you eliminate the 

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 11/21/17 11:34 AM, Shawn Heisey wrote:
> On 11/16/2017 5:21 PM, Phil Steitz wrote:
>> That should not be possible, as once a connection has been checked
>> out, it is not eligible for eviction.  And even if it were evicted
>> by the pool, the pool would set the DelegatingConnection's closed
>> property to true so isValid would return false rather than
>> throwing.  It looks like the situation here is that the pool thinks
>> the connection is open, but the underlying physical connection has
>> been closed.
> I am very glad to hear you describe the intended operation this way --
> the gist of which seems to say that when getConnection is called, it
> *should* immediately mark that connection as ineligible for eviction.
>
> My current suspicion is that there is a bug, where getConnection isn't
> marking the connection as ineligible like it should, or maybe where
> there is insufficient synchronization to prevent the eviction thread
> from closing/evicting the connection before getConnection *does* mark
> the connection as ineligible.  This is just a theory, but it does fit
> what I've seen.  I have tried to investigate the code, but the project
> has used object inheritance and abstraction very effectively, which
> makes it difficult for me to follow exactly what the code is doing,
> because I am not familiar with all of that inheritance.

It's always possible that there is a bug. The relevant code is in
commons pool.  The code to look at is in
o.a.c.pool2.impl.GenericObjectPool#evict.  The guard is here:

if (!underTest.startEvictionTest()) {
    // Object was borrowed in another thread
    // Don't count this as an eviction test so reduce i;
    i--;
    continue;
 }

The objects that the pool works with are PooledObjects.  DBCP's
PoolableConnectionFactory creates DefaultPooledObjects, which
implement the method above by returning false unless the object is
idle in the pool.  If it is idle in the pool, its state is changed
to indicate that it is under test.  This method is synchronized (on
the PooledObject's monitor), as are the other methods that read or
mutate its PooledObjectState.

Before returning an object from the pool to a client,
GenericObjectPool's borrowObject calls the PooledObject's allocate
method.  That method, again synchronized on the object, checks to
make sure that the object's state is idle in the pool.  If so, it
changes its state to allocated and before returning it to the client.

As I said above, it's always possible that there is a bug somewhere,
but the setup is simple and it has been hammered pretty hard for a
few years now.
>
> I am not aware of any way for one thread in my code to close a
> connection obtained by another thread.  My design intent is to ensure
> that this is completely impossible, by handling the full lifecycle of a
> connection in one thread and never passing connections between threads. 
> I am not aware of any places in the program that violate this design
> intent ... but I've written plenty of bugs in my time, so it's always
> possible that I *have* made a mistake like that.

As I said in my first response, the most common explanation for this
kind of exception when using DBCP is that the underlying physical
connection is closed on the server (or network) side without DBCP
knowing about it.  And the most common cause of that is server-side
idle connection timeout.

It just occurred to me that since you do not have any of the testXxx
flags set to true, DBCP is never actually exercising the
connections.  Is it possible that sometimes your code checks out a
connection from the pool but does not use it?  "Idle" to DBCP/Pool
means idle in the pool - i.e., not checked out to a client.  So if
connections are checked out and checked back in without being used,
they will not be evicted, but could be timed out by the server.

Phil
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 11/17/17 3:27 PM, Phil Steitz wrote:
> On 11/17/17 12:22 PM, Shawn Heisey wrote:
>> On 11/16/2017 5:21 PM, Phil Steitz wrote:
>>>> How can a connection that I *just* retrieved from the pool be closed?
>>> Should not happen.  In most cases like this, it turns out not to be
>>> the pool that closed the connection.  The most common causes of this
>>> are network connectivity problems or the server closing the
>>> connection due to idle timeout.  The latter seems unlikely given
>>> that you have idle eviction configured.  Is it possible that some
>>> clients are holding connections for a very long time without using
>>> them?  The evictor only acts on connections that are idle in the
>>> pool (i.e. it does not do anything to connections that are checked
>>> out to clients).
>> Thank you for your reply.
>>
>> My program does its work once a minute, and I had also configured the
>> idle eviction to run once a minute.
>>
>> Just to see what it would do, I slightly changed the run interval for
>> the idle eviction -- for one of the object pools, adding 943
>> milliseconds, and 944 milliseconds for the other.  I've only seen the
>> problem happen once since making that change, where before it would
>> happen several times per day.  This isn't actual evidence, but it is
>> curious.
> See comment below on the config.
>> Although a network problem is always a possibility, it's not likely. 
>> It's a gigabit LAN with Cisco switches.  I will trace the network path
>> and check all the switch logs to see whether that might indeed be the
>> problem.
>>
>>> That should not be possible, as once a connection has been checked
>>> out, it is not eligible for eviction.  And even if it were evicted
>>> by the pool, the pool would set the DelegatingConnection's closed
>>> property to true so isValid would return false rather than
>>> throwing.  It looks like the situation here is that the pool thinks
>>> the connection is open, but the underlying physical connection has
>>> been closed.
>> The code is within a try/catch block that traps all exceptions, so I am
>> 100 percent certain that isValid is not throwing an exception, it's just
>> returning false. 
> OK, sorry I misread your initial post.  That means either the pool
> agrees that the connection is closed (which would point to a pool or
> DBCP bug) or the driver's isValid() on the underlying physical
> connection is returning true (without throwing).
Sorry, I meant "false" above.  DelegatingConnection's isValid is just

public boolean isValid(int timeout) throws SQLException {
    if (isClosed()) {
    return false;
    }
    try {
    return _conn.isValid(timeout);
    }
    catch (SQLException e) {
    handleException(e);
    return false;
    }
    }

Note that the validation query is not executed here.  That query is
only used by the pool lifecycle methods when you have testOnBorrow,
testOnReturn or testWhileIdle set to true.  handleExeption as
overridden by PoolableConnection rethrows.  _conn is the underlying
physical connection,

Phil

>   If it's not too
> hard to do, it would be good to also log what comes back from
> isClosed() on the DelegatingConnection (what dsMaster returns).  If
> that returns true, then either this is a DBCP or pool bug or you
> have somehow closed the DelegatingConnection via another reference
> or something.  If it returns false, that means the pool thinks the
> connection is open so it is unlikely that the pool closed it.
>>  When I first started investigating my strange alarms,
>> I couldn't tell what was wrong at all until I decided to add some code
>> to try a query on the connection that said it wasn't valid.   I fully
>> expected that query to fail, I was just doing it to try and discover why
>> isValid returned false.
>>
>> This is a very recent problem, but the code hasn't had any changes since
>> well before the problem began.  That does lend credibility to the
>> possibility of a network problem, which as I said, I will investigate.
>>
>>> It would be good to see your full pool config and a stack trace of
>>> the exception above.  You might also look at the server logs to see
>>> what is happening on the Mysql side.
>> Tell me exactly what I need to do in order to gather the pool config
>> you're after.  Are you looking for Java code, or some kind of
>> information from the objects that I need to gather?  Here's the code
>> that defines a datasource. 
> What you have below is fine.  I just wanted to see the pool
> configuration settings.

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 11/17/17 12:22 PM, Shawn Heisey wrote:
> On 11/16/2017 5:21 PM, Phil Steitz wrote:
>>> How can a connection that I *just* retrieved from the pool be closed?
>> Should not happen.  In most cases like this, it turns out not to be
>> the pool that closed the connection.  The most common causes of this
>> are network connectivity problems or the server closing the
>> connection due to idle timeout.  The latter seems unlikely given
>> that you have idle eviction configured.  Is it possible that some
>> clients are holding connections for a very long time without using
>> them?  The evictor only acts on connections that are idle in the
>> pool (i.e. it does not do anything to connections that are checked
>> out to clients).
> Thank you for your reply.
>
> My program does its work once a minute, and I had also configured the
> idle eviction to run once a minute.
>
> Just to see what it would do, I slightly changed the run interval for
> the idle eviction -- for one of the object pools, adding 943
> milliseconds, and 944 milliseconds for the other.  I've only seen the
> problem happen once since making that change, where before it would
> happen several times per day.  This isn't actual evidence, but it is
> curious.

See comment below on the config.
>
> Although a network problem is always a possibility, it's not likely. 
> It's a gigabit LAN with Cisco switches.  I will trace the network path
> and check all the switch logs to see whether that might indeed be the
> problem.
>
>> That should not be possible, as once a connection has been checked
>> out, it is not eligible for eviction.  And even if it were evicted
>> by the pool, the pool would set the DelegatingConnection's closed
>> property to true so isValid would return false rather than
>> throwing.  It looks like the situation here is that the pool thinks
>> the connection is open, but the underlying physical connection has
>> been closed.
> The code is within a try/catch block that traps all exceptions, so I am
> 100 percent certain that isValid is not throwing an exception, it's just
> returning false. 

OK, sorry I misread your initial post.  That means either the pool
agrees that the connection is closed (which would point to a pool or
DBCP bug) or the driver's isValid() on the underlying physical
connection is returning true (without throwing).  If it's not too
hard to do, it would be good to also log what comes back from
isClosed() on the DelegatingConnection (what dsMaster returns).  If
that returns true, then either this is a DBCP or pool bug or you
have somehow closed the DelegatingConnection via another reference
or something.  If it returns false, that means the pool thinks the
connection is open so it is unlikely that the pool closed it.
>  When I first started investigating my strange alarms,
> I couldn't tell what was wrong at all until I decided to add some code
> to try a query on the connection that said it wasn't valid.   I fully
> expected that query to fail, I was just doing it to try and discover why
> isValid returned false.
>
> This is a very recent problem, but the code hasn't had any changes since
> well before the problem began.  That does lend credibility to the
> possibility of a network problem, which as I said, I will investigate.
>
>> It would be good to see your full pool config and a stack trace of
>> the exception above.  You might also look at the server logs to see
>> what is happening on the Mysql side.
> Tell me exactly what I need to do in order to gather the pool config
> you're after.  Are you looking for Java code, or some kind of
> information from the objects that I need to gather?  Here's the code
> that defines a datasource. 

What you have below is fine.  I just wanted to see the pool
configuration settings.  Looks OK, but might end up creating a lot
of connection churn having maxIdle set to 6 with maxTotal at 30.  I
guess the idea is that when it runs, you want to have an idle
connection available for each shard with the ability to add more up
to a total of 30, but you don't want those idle connections to stay
in the pool when the work isn't being done.  Do you know how many
connections actually get opened during the work period?  If that is
more than a few more than 6, you are kind of defeating the purpose
of the pool by closing them all when they get returned (as the
maxIdle setting will force).

The fact that changing the eviction interval to not exactly coincide
with when the work happens makes the problem go away is troubling. 
I can see how that might improve performance, but it does make the
pool bug scenario more likely.

One more question on the config.  Why are you running the evictor
every minute and using eviction to close idle connections?  The
maxIdle setting

Re: [DBCP] Connection just obtained from datasource is invalid

[Phil Steitz]

On 11/15/17 12:50 PM, Shawn Heisey wrote:
> I'm using DBCP 2.1.1 with POOL 2.4.3 (which are the latest releases as
> of a few minutes ago) to handle connections to MySQL databases.  In my
> program I have code like this:
>
>   conn = ds.getConnection();
>   conn.clearWarnings();
>   if (conn.isValid(5))
>
> Recently, I've been having problems where the connection is showing up
> as *not* valid, and I couldn't figure out why.  My code uses
> conn.getWarnings() on invalid connections to try and determine why the
> connection is invalid, but it wasn't getting anything.  No exception was
> being logged, so the return was likely null.  The validation query is
> "SELECT 1".
>
> After scratching my head for a while, I decided to have the code run a
> "SELECT 1" query on the connection that reports invalid.  As expected
> because the connection tests as invalid, that query failed, but the
> specific failure was surprising:
>
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
> operations allowed after connection closed.
>
> How can a connection that I *just* retrieved from the pool be closed?

Should not happen.  In most cases like this, it turns out not to be
the pool that closed the connection.  The most common causes of this
are network connectivity problems or the server closing the
connection due to idle timeout.  The latter seems unlikely given
that you have idle eviction configured.  Is it possible that some
clients are holding connections for a very long time without using
them?  The evictor only acts on connections that are idle in the
pool (i.e. it does not do anything to connections that are checked
out to clients).
> d
>
> The only thing that comes to mind is that I have configured idle
> connection eviction:
>
>         opMaster.setTimeBetweenEvictionRunsMillis(Const.ONE_MINUTE);
>         opMaster.setMinEvictableIdleTimeMillis(Const.ONE_MINUTE * 5);
>
> If the idle connection eviction is what's to blame for this (connection
> becomes invalid between the time it is  retrieved and the validity is
> checked), it seems like that shouldn't be possible.

That should not be possible, as once a connection has been checked
out, it is not eligible for eviction.  And even if it were evicted
by the pool, the pool would set the DelegatingConnection's closed
property to true so isValid would return false rather than
throwing.  It looks like the situation here is that the pool thinks
the connection is open, but the underlying physical connection has
been closed.

It would be good to see your full pool config and a stack trace of
the exception above.  You might also look at the server logs to see
what is happening on the Mysql side.

Phil
>
> Before I open an issue in Jira, I would like to know if there's anything
> I can add to my code (logging, info retrieval, etc) to check what might
> be going wrong -- try to confirm my theory that idle connection eviction
> is to blame.  If that is indeed the problem, I would comment that
> getConnection() on the datasource should probably reset the idle timer
> so that the connection will not be quickly closed/pruned.


>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] outdated and confusing documentation

[Phil Steitz]

On 7/31/17 9:07 AM, Steve Cohen wrote:
> The DBCP package description page is referred to from the Overview page
> as containing important information on usage of the package.
>
> The following sample code is provided there:
>
>
>> GenericObjectPool connectionPool = new GenericObjectPool(null);
>> ConnectionFactory connectionFactory = new 
>> DriverManagerConnectionFactory("jdbc:some:connect:string", "username", 
>> "password");
>> PoolableConnectionFactory poolableConnectionFactory = new 
>> PoolableConnectionFactory(connectionFactory,connectionPool,null,null,false,true);
>> PoolingDataSource dataSource = new PoolingDataSource(connectionPool);
>
>
> The third line appears out of date:  No such constructor of
> PoolableConnectionFactory appears to exist.  Instead I find
>
> PoolableConnectionFactory(ConnectionFactory connFactory, ObjectName
> dataSourceJmxName)
>
> OK, I can use that.
>
> But ...
> Eclipse is warning me that this PoolableConnectionFactory is not used in
> the code.  And it's right!  Nothing ties the poolableConnectionFactory
> object we create in that line to anything else, and it doesn't appear to
> be needed.
>
> At any rate, I'm confused.  What role does this object play in the
> scenario the documentation is referring to?

The package javadoc was not updated to reflect the API changes made
in dbcp2.  It would be good to raise a JIRA for this. 
Unfortunately, the examples link appears now to be broken, but there
is an example in the /doc directory of the source distribution that
shows how to set up a PoolingDataSource.  You can also see this in
the unit tests.  Here is the relevant bit of the sample code from
the /doc sources:

 public static DataSource setupDataSource(String connectURI) {
//
// First, we'll create a ConnectionFactory that the
// pool will use to create Connections.
// We'll use the DriverManagerConnectionFactory,
// using the connect string passed in the command line
// arguments.
//
ConnectionFactory connectionFactory =
new DriverManagerConnectionFactory(connectURI,null);

//
// Next we'll create the PoolableConnectionFactory, which wraps
// the "real" Connections created by the ConnectionFactory with
// the classes that implement the pooling functionality.
//
PoolableConnectionFactory poolableConnectionFactory =
new PoolableConnectionFactory(connectionFactory, null);

//
// Now we'll need a ObjectPool that serves as the
// actual pool of connections.
//
// We'll use a GenericObjectPool instance, although
// any ObjectPool implementation will suffice.
//
ObjectPool connectionPool =
new GenericObjectPool<>(poolableConnectionFactory);
   
// Set the factory's pool property to the owning pool
poolableConnectionFactory.setPool(connectionPool);

//
// Finally, we create the PoolingDriver itself,
// passing in the object pool we created.
//
PoolingDataSource dataSource =
new PoolingDataSource<>(connectionPool);

return dataSource;
}

hth,

Phil
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: Could DBCP set different username when getConnection

[Phil Steitz]

On 2/16/17 1:31 AM, John Patrick wrote:
> >From my understanding of you question. You would need a pool per role/user.

That is available in PerUserPoolDataSource [1].

Phil

[1] http://commons.apache.org/proper/commons-dbcp/api-2.1.1/index.html

Note to devs:  the 2.2 javadocs seem to be missing from the web site.
>
> John
>
>
> On 16 February 2017 at 08:48, 王钦  wrote:
>> Hi All:
>>
>> I define some roles and grant different privileges in database. And I
>> use DBCP to manage the connection. How could I set different user name when
>> I get one connection from DBCP?
>>
>> Thanks for reading this mail.
>>
>>
>> Qin
>>
>>
>>
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
>> For additional commands, e-mail: user-h...@commons.apache.org
>>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [EXTERNAL]Re: DBCP issuing more sessions than maxActive sessions in Oracle

[Phil Steitz]

On 1/24/17 4:55 PM, Romero, René wrote:
> We're running  commons-dbcp-1.4 and commons-pool-1.6. To me, it sounds like 
> we're being impacted by this bug:
>
> https://issues.apache.org/jira/browse/DBCP-379
>
> [DBCP-379] number of connections created has crossed more 
> ...<https://issues.apache.org/jira/browse/DBCP-379>
> issues.apache.org
> I found that when the minIdle is configured then during loads, common pool 
> creates more number of connections it will be greater than maxActive.
>
> Is it transparent application-wise to jump from these versions to the 
> latests? Or what upgrading path do you suggest we take?
>
>
> Thanks again.
>
>
>
> Rene Romero Benavides
> Administrador de Bases de Datos para Command Center
> +52(55) 11020460 Ext. 6432 |  Office
>
> +52 1 5545664012 |  Mobile
>
> www.omnitracs.com/mx<http://www.omnitracs.com/mx>
>
> Register today!  www.omnitracs.com/outlook<http://www.omnitracs.com/outlook>
>
> <https://outlook.office.com/owa/service.svc/s/GetFileAttachment?id=AAMkAGM2NWIyZGFmLWZhMTYtNDUzNC04YzQ5LWEwOTA1Zjk4MTY3ZgBGAABp9rw%2F5DvqQ5VNu%2FP5FANvBwCC6JFKXcS6RJy5Q76IhGGqAAEMAACC6JFKXcS6RJy5Q76IhGGqAACGtmA4AAABEgAQANT3RQAH0VJIrveZGngUUTQ%3D=zgTt0gEIPEaHc-Q35K2xD7ADDSE3yNIYLGPJUeTkGumastS1fdcpigPbw4ASyo9SSzPAsJtdi6o>
> 
> From: Phil Steitz <phil.ste...@gmail.com>
> Sent: Tuesday, January 24, 2017 3:43:29 PM
> To: Commons Users List
> Subject: [EXTERNAL]Re: DBCP issuing more sessions than maxActive sessions in 
> Oracle
>
> What versions of dbcp and pool are you running?
>
> Looks like 1.x from the package names.  The first thing to look at
> is the changelogs[1] for releases since the pool and dbcp versions
> that you are running.
>
> Phil
>
> [1] for dbcp this is
> https://commons.apache.org/proper/commons-dbcp/changes-report.html
> There is a similar page on the pool site.
>
> On 1/24/17 3:02 PM, Romero, René wrote:
>> Hello guys.
>>
>> Under what circumstances could we get more than MaxActive sessions in the 
>> database if DBCP is our only means of access to it?
>>
>> This is our current configuration and it's surpassing 160 Oracle sessions at 
>> times when it shouldn't go beyond 32 (our current value for MaxActive).
>>
>>
>> DB = Oracle 11g
>>
>> Data access via spring famework.
>>
>>
>> Thanks in advance.
>>
>>
>>   > p:url="${dw.db.url}" p:username="${dw.db.username}"
>> p:password="${dw.db.password}" 
>> p:driverClassName="${dw.db.driverClassName}" init-method="createDataSource" 
>> destroy-method="close">
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>   
>>
>>
>> dw.db.url=jdbc:oracle:thin:someDB/somePass@someHost:1522:SomeInstance
>> dw.db.username=someUser
>> dw.db.password=somePass
>> dw.db.driverClassName=oracle.jdbc.OracleDriver
>> dw.db.allowed.operations=select
>> dw.db.schema=1|COMMON_APP_VIEWS_01, 81|COMMON_APP_VIEWS_81
>> dw.db.maxRows=500
>> dw.db.type=ORACLE
>> dw.db.validation.query=select 1 from dual
>> dw.db.initial.connection.size=4
>> dw.db.max.connection.size=32
>> dw.db.minIdle.connections=4
>> dw.db.maxIdle.connections=4

The settings above will create a lot of connection churn and may be
what is causing you to hit an old pool or dbcp bug.  When you set
minIdle = maxIdle, you are telling the pool that it needs to try to
keep *exactly* that number (in your case 4) idle connections at all
times.  The maxIdle setting means that when there are already 4 idle
connections and a 5th is returned, it is destroyed.  If the idle
count subsequently drops below that, since you have the evictor
enabled, it will try to ensureMinIdle when it runs, creating new
ones.  In general, it is better to have maxIdle >> minIdle.  This
applies to dbcp2 as well.

Phil
>>
>>
>> What other info would you need?
>>
>>
>> Rene Romero Benavides
>> Administrador de Bases de Datos para Command Center
>> +52(55) 11020460 Ext. 6432 |  Office
>>
>> +52 1 5545664012 |  Mobile
>>
>> www.omnitracs.com/mx<http://www.omnitracs.com/mx<http://www.omnitracs.com/mx<http://www.omnitracs.com/mx>>
>>
>> Register today!  
>> www.omnitracs.com/outlook<http://www.omnitracs.com/outlook<http://www.omnitracs.com/outlook<http://www.omnitracs.com/outlook>>
>>
>> <https://outlook.office.com/owa/service.svc/s/GetFileAttachment?id=AAMkAGM2NWIyZGFmLWZhMTYtNDUzNC04YzQ5LWEwOTA1Zjk4MTY3ZgBGAABp9rw%2F5DvqQ5VNu%2FP5FANvBwCC6JFKXcS6RJy5Q76IhGGqAAEMAACC6JFKXcS6RJy5Q76IhGGqAACGtmA4AAABEgAQANT3RQAH0VJIrveZGngUUTQ%3D=zgTt0gEIPEaHc-Q35K2xD7ADDSE3yNIYLGPJUeTkGumastS1fdcpigPbw4ASyo9SSzPAsJtdi6o>
>>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
> This message was received from outside of the Omnitracs network
>



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: DBCP issuing more sessions than maxActive sessions in Oracle

[Phil Steitz]

What versions of dbcp and pool are you running?

Looks like 1.x from the package names.  The first thing to look at
is the changelogs[1] for releases since the pool and dbcp versions
that you are running.

Phil

[1] for dbcp this is
https://commons.apache.org/proper/commons-dbcp/changes-report.html
There is a similar page on the pool site.

On 1/24/17 3:02 PM, Romero, René wrote:
> Hello guys.
>
> Under what circumstances could we get more than MaxActive sessions in the 
> database if DBCP is our only means of access to it?
>
> This is our current configuration and it's surpassing 160 Oracle sessions at 
> times when it shouldn't go beyond 32 (our current value for MaxActive).
>
>
> DB = Oracle 11g
>
> Data access via spring famework.
>
>
> Thanks in advance.
>
>
>p:url="${dw.db.url}" p:username="${dw.db.username}"
> p:password="${dw.db.password}" 
> p:driverClassName="${dw.db.driverClassName}" init-method="createDataSource" 
> destroy-method="close">
> 
> 
> 
> 
> 
> 
> 
>   
>
>
> dw.db.url=jdbc:oracle:thin:someDB/somePass@someHost:1522:SomeInstance
> dw.db.username=someUser
> dw.db.password=somePass
> dw.db.driverClassName=oracle.jdbc.OracleDriver
> dw.db.allowed.operations=select
> dw.db.schema=1|COMMON_APP_VIEWS_01, 81|COMMON_APP_VIEWS_81
> dw.db.maxRows=500
> dw.db.type=ORACLE
> dw.db.validation.query=select 1 from dual
> dw.db.initial.connection.size=4
> dw.db.max.connection.size=32
> dw.db.minIdle.connections=4
> dw.db.maxIdle.connections=4
>
>
> What other info would you need?
>
>
> Rene Romero Benavides
> Administrador de Bases de Datos para Command Center
> +52(55) 11020460 Ext. 6432 |  Office
>
> +52 1 5545664012 |  Mobile
>
> www.omnitracs.com/mx
>
> Register today!  www.omnitracs.com/outlook
>
> 
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: I need to close the connection, not return it back to the Connection Pool

[Phil Steitz]

On 12/22/16 7:22 PM, 王钦 wrote:
> Hi Shawn & Phil:
>
> Thank you for your help.
>
> The scenario I faced is like this:
>
> In the usual work time, there will be plenty of connections to
> the Impala Database. So I need a Connection Pool to manage them.
>
> When a query is executed for a long time and the client wants
> to cancel it manually, closing the connection for this query
> really is one solution (or maybe a poor solution.)

OK, so the best solution would be to find a way to kill or set a
timeout on the query, but I understand that may not be possible, so
you want to be more brutal and kill the connection.  The best here
would be to upgrade to DBCP 2.2 and use the invalidate method that
was added for this kind of use case.  Upgrading to DBCP2/Pool2 has
other benefits as well.  

Phil
>
> And thank you for your help again.
>
>
> Best Wishes
>
> Qin
>
>
>
> On 2016年12月23日 04:47, Phil Steitz wrote:
>> On 12/22/16 9:34 AM, Shawn Heisey wrote:
>>> On 12/22/2016 4:49 AM, 王钦 wrote:
>>>>  When I use DBCP-1.4 in my work, I need to close the
>>>> connection
>>>> which is generated by BasicDataSource. How can I do it? Close the
>>>> connection, while not returning it back to the Connection Pool.
>>> The PoolableConnection class (which I believe is the actual type of
>>> object you will receive from the data source) has a
>>> "reallyClose()" method.
>>>
>>> http://commons.apache.org/proper/commons-dbcp/api-1.4/org/apache/commons/dbcp/PoolableConnection.html#reallyClose()
>>>
>>>
>>> I'm no expert, but I think you might be able to cast the connection
>>> received from BasicDataSource to the Poolable variety.
>>>
>>> If you intend to "reallyClose()" every connection (rather than
>>> just some
>>> of them), I do have to wonder why you're using DBCP at all. 
>>> Obtaining
>>> connections from JDBC directly and closing them normally would
>>> accomplish the same thing without a code dependency and slightly
>>> less
>>> overhead.
>> I agree with your statement about defeating the purpose of the pool
>> if you do this every time.  If done only when the client knows the
>> connection is bad, the solution above will unfortunately leak pool
>> capacity (reallyClose doesn't inform the pool that the checked out
>> connection is never coming back).  A sort of ugly workaround for
>> that would be to enable abandoned connection cleanup, which would
>> eventually clean these up.
>>
>> What you need to do to do this cleanly is to get the underlying
>> GenericObjectPool to invalidate the PoolableConnection so its
>> accounting is correctly updated.  If you are willing to upgrade to
>> DBCP 2, BasicDataSource now has an invalidateConnection method that
>> handles this.  If not, the only way to do this without leaking
>> capacity is to do what the source for that method does, which
>> requires that you get a reference to the underlying object pool.
>> BasicDatasource does not expose that, so if you want to go this
>> route, you need to manually create the GOP and use a
>> PoolingDatasource.
>>
>> Phil
>>> Thanks,
>>> Shawn
>>>
>>>
>>> -
>>>
>>> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
>>> For additional commands, e-mail: user-h...@commons.apache.org
>>>
>>>
>>
>>
>> -
>>
>> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
>> For additional commands, e-mail: user-h...@commons.apache.org
>>
>>
>
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: I need to close the connection, not return it back to the Connection Pool

[Phil Steitz]

On 12/22/16 9:34 AM, Shawn Heisey wrote:
> On 12/22/2016 4:49 AM, 王钦 wrote:
>> When I use DBCP-1.4 in my work, I need to close the connection
>> which is generated by BasicDataSource. How can I do it? Close the
>> connection, while not returning it back to the Connection Pool. 
> The PoolableConnection class (which I believe is the actual type of
> object you will receive from the data source) has a "reallyClose()" method.
>
> http://commons.apache.org/proper/commons-dbcp/api-1.4/org/apache/commons/dbcp/PoolableConnection.html#reallyClose()
>
> I'm no expert, but I think you might be able to cast the connection
> received from BasicDataSource to the Poolable variety.
>
> If you intend to "reallyClose()" every connection (rather than just some
> of them), I do have to wonder why you're using DBCP at all.  Obtaining
> connections from JDBC directly and closing them normally would
> accomplish the same thing without a code dependency and slightly less
> overhead.

I agree with your statement about defeating the purpose of the pool
if you do this every time.  If done only when the client knows the
connection is bad, the solution above will unfortunately leak pool
capacity (reallyClose doesn't inform the pool that the checked out
connection is never coming back).  A sort of ugly workaround for
that would be to enable abandoned connection cleanup, which would
eventually clean these up. 

What you need to do to do this cleanly is to get the underlying
GenericObjectPool to invalidate the PoolableConnection so its
accounting is correctly updated.  If you are willing to upgrade to
DBCP 2, BasicDataSource now has an invalidateConnection method that
handles this.  If not, the only way to do this without leaking
capacity is to do what the source for that method does, which
requires that you get a reference to the underlying object pool. 
BasicDatasource does not expose that, so if you want to go this
route, you need to manually create the GOP and use a PoolingDatasource.

Phil
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp]the activity when number of active connections reach (maxActive-minIdle)

[Phil Steitz]

On Fri, Jan 8, 2016 at 10:27 PM, sum2000  wrote:

> for example,pool size:
> db.pool.size.init=5
> db.pool.size.active.max=20
> db.pool.size.idle.max=20
> db.pool.size.idle.min=10
> when # of active connections reach 10, what's the following activity of
> dbcp?
> We know it can’t create new one for idle due to no availabe pool resource,
> if true dbcp can’t keep minIdle for 10, so what's next happen?
>

The maxTotal [1] property is a hard limit on the total number of
connections that can be open under management by the pool at a given time
(checked out or idle).  The minIdle property really expresses an objective,
not a guarantee.  In the situation you describe, the number of idle
connections will drop down below minIdle.

Another thing to note is that as of version 2.1 of DBCP, as indicated in
the javadoc for minIdle, the pool only attempts to ensure minIdle when the
evictor runs, so unless you have timeBetweenEvictionRunsMillis set to a
positive value, there will be no attempt by the pool to ensure minIdle.

Phil

[1] In DBCP 1.x, this property is named "maxActive".

>
>
> Thank you.
> Yong Fu

Re: [math] Summary Stats Higher Moments?

[Phil Steitz]

On 12/22/15 9:58 AM, michael.brzustow...@gmail.com wrote:
> Hi,
>
> I see that org.apache.commons.math3.stat.descriptive.DescriptiveStatistics
> uses the singleton update formulas (from Pebay) for calculating
> (un-normalized) moments up to the 4th moment. Is there some reason that
> org.apache.commons.math3.stat.descriptive.SummaryStatistics excludes both
> third and fourth central moments?
>
> Is it just a matter of computational efficiency, ie. DescriptiveStatistics
> calculates moments only when the getter is invoked (and all orders need not
> be calculated at once) while the "storeless" SummaryStatistics would need
> to calculate all 4 orders at every call to update()?

Yes, that is the reason; but it is really more a matter of no one
having asked for this feature.  You are correct that the updating
formulas make this possible and the nested nature of the moments
means that there should not be much cost to adding the third and
fourth moments.  I would be happy to review and apply a patch (with
tests) adding these.

Phil

>  Or is there some other
> blocker?
>
> Thanx,
> Mike Brzustowicz
>



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [MATH] Why no o.a.c.math3.stat.descriptive.moment.QuadraticMean?

[Phil Steitz]

On 10/8/15 2:00 PM, Evan Meagher wrote:
> An application I'm writing has a need to calculate RMS values for series of
> numbers. I can achieve this via `SummaryStatistics.getQuadraticMean`, but
> I'm left wondering why there isn't a corresponding `UnivariateStatistic`
> subclass? It seems to me that a QuadraticMean class within the
> org.apache.commons.math3.stat.descriptive.moment
> package
> 
> would be a reasonable addition to commons-math3.
>
> Would be willing to implement one if there's agreement that it's worth
> introducing.

+1 to add this class for consistency.  Thanks in advance for a patch
with test cases.

Phil
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] setting properties

[Phil Steitz]

On 10/8/15 12:00 PM, Pruitt, Byron S wrote:
> The BasicDataSource class has methods for setting some validation policies
>
> getTestOnBorrow())
> getTestOnCreate())
> getTestOnReturn())
> getTestWhileIdle())
>
> How are these invoked programmatically?  I started with the 
> PoolingDataSourceExample code and nothing created can set those.

To use those properties, you need to instantiate a BasicDataSource,
which wraps a PoolingDataSource.

Phil
>
> Thanks.
>
> -SP
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] auto reconnecting

[Phil Steitz]

On 10/7/15 12:35 PM, Pruitt, Byron S wrote:
> My server has long standing idle periods.  We are seeing:  
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet 
> successfully received from the server was...  errors.
>
> I would like to fix it as simply as possible.  MySQL has two properties 
> autoReconnect and autoReconnectForPools that might fix the problem.  They 
> both default to false and I can't find an easy way to set them.
> I had an earlier problem with auto commit and I fixed it with the 
> setDefaultAutoCommit method on PoolableConnectionFactory.  But, it doesn't 
> have a similar setter for auto reconnect.
>
> The only example I found of setting the autoReconnect property is on the 
> connection url.  I use the interface where I pass an instance of Properties, 
> so I assume I can set those properties with it.
>
> I am curious if there are better ways to set those properties.

If you are using BasicDataSource, you can use it's
addConnectionProperty method to add driver-specific connection
properties such as this.  Just make sure to add the property before
you use the datasource.

You can achieve the same effect by enabling validation on borrow or
while idle (testOnBorrow, testWhileIdle).

Phil
>
> Thanks.
>
> -SP
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] inconsistently set autocommit

[Phil Steitz]

On 9/16/15 10:35 AM, Pruitt, Byron S wrote:
> My usual process for learning something is a quick prototype.  For setting up 
> a driver pool with autocommit = false, I implemented the little quickie 
> prototype below.
>
> Properties dbProp = new Properties();
> dbProp.put("user", "aUser");
> dbProp.put("password", "aPw");
> dbProp.put("loginTimeout", "35");
> dbProp.put("binaryBatchInsert", "true");
> dbProp.put("autocommit", false);
> DataSource dataSource = setupDataSource("myConnectURI", dbProp);  //this the 
> standard example on the website
>
> Connection conn = dataSource.getConnection();
> System.out.println("autocommit: " + conn.getAutoCommit());   //this prints 
> out false
>
> Next.
> I moved the above code verbatim to my actual server code.
> The first connection I get from the pool has autocommit == false.  I return 
> this connection back to the pool.
> But the second time I get a connection from the pool it has autocommit == 
> true.
>
> I have no idea why autocommit is set to a different state for the second 
> connection?  I can do a test and then set to autocommit = false, but I would 
> like to understand the inconsistency.

Great question.  By default, DBCP's PoolableConnectFactory sets
autocommit to true when it returns a connection to the pool.  See
DBCP-97 for the rationale for this default behavior.  This behavior
can be modified by setting the enableAutoCommitOnReturn of
BasicDataSource or PoolableConnectionFactory to false.  You can also
use the defaultXxx properties of BDS or PCF to set default
properties for connections supplied by the pool.  These properties
ensure that connections will be restored to a consistent state when
the are checked out from the pool, even if client code has mutated
them while the connection has been checked out.

Phil
>
> Thanks.
>
> Steve Pruitt
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] new user questions

[Phil Steitz]

> On Sep 3, 2015, at 9:22 AM, Pruitt, Byron S  wrote:
> 
> I am new to the DBCP framework and I have some newbie questions.
> 
> I have the PoolingDataSourceExample working with my driver and I get 
> connections from my DataSource.
> 
> What I am unclear on is how I return a connection back to the pool.
> After I have finished with executing a statement and finished with the 
> ResultSet or having issued a commit if that is needed, then what?
> Do I do nothing with the connection?  How is the connection returned to pool, 
> so it can be reused?

Just call close() on the connection.  That will cause the physical connection 
to be returned to the pool.
> 
> 
> Also, it looks like the way to configure the pool for the number of 
> connections, etc. is via GenericObjectPoolConfig passed to the 
> GenericObjectPool instance.
> 
> 
Correct.  If you don't want to separately create and configure the object pool, 
have a look at BasicDatasource, which exposes the pool configure parameters 
directly.

Phil
> 
> Thanks ahead for any answers.
> 
> 
> Steve Pruitt
> 

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] setting connection defaults

[Phil Steitz]

On 9/3/15 12:26 PM, Pruitt, Byron S wrote:
> What is the best to set properties like auto commit.  The DataSource 
> interface doesn't expose anything.  The only thing I can find that supports 
> these properties is BasicDataSource.

This depends on your application, but it's good to start with the
defaults, which as of DBCP 2.x are the defaults set by the database
driver.  You can change the default connection properties by
specifying them in the BasicDataSource config or by setting them on
the PoolableConnectionFactory if you are manually creating the
PoolingDataSource.

Phil
>
> Thanks in advance.
>
>
> Steve Pruitt
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

[ANNOUNCEMENT] Apache Commons DBCP 2.1.1 released

[Phil Steitz]

The Apache Commons Team is pleased to announce the release of Apache Commons 
DBCP 2.1.1.

The Apache Commons DBCP open source software library implements database 
connection pooling. Version 2.1.1 is a patch release, including bug fixes only. 
Full details of all the changes in version 2.1.1 can be found in the changelog:
http://commons.apache.org/dbcp/changes-report.html

Source and binary distributions are available for download from the Apache 
Commons download site:
http://commons.apache.org/dbcp/download_dbcp.cgi

When downloading, please verify signatures using the KEYS file available at the 
above location.

For complete information on Commons DBCP, including instructions on how to 
submit bug reports, patches, or suggestions for improvement, see the Apache 
Commons DBCP website:
http://commons.apache.org/dbcp

Phil Steitz, on behalf of the Apache Commons community



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

[ANNOUNCEMENT] Apache Commons Pool 2.4.2 released

[Phil Steitz]

The Apache Commons Team is pleased to announce the release of Apache Commons 
Pool 2.4.2.

The Apache Commons Pool open source software library provides an object-pooling 
API and a number of object pool implementations.

No client code changes are required to migrate from any of the 2.x versions to 
2.4.2. Users of version 1.x should consult the migration guide on the Commons 
Pool web site.

Source and binary distributions are available for download from the Apache 
Commons download site:
http://commons.apache.org/pool/download_pool.cgi

When downloading, please verify signatures using the KEYS file available at the 
above location.

Full details of all the changes in 2.4.2 can be found in the changelog:
http://commons.apache.org/pool/changes-report.html

For complete information on Commons Pool, including instructions on how to 
submit bug reports, patches, or suggestions for improvement, see the Apache 
Commons Pool website:
http://commons.apache.org/pool

Phil Steitz, on behalf of the Apache Commons community



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Fwd: Re: [dbcp] Abandon Connection - not Logging in standalone

[Phil Steitz]

Forgot to copy list


 Forwarded Message 
Subject:Re: [dbcp] Abandon Connection - not Logging in standalone
Date:   Sun, 26 Jul 2015 17:02:38 -0700
From:   Phil Steitz phil.ste...@gmail.com
To: Bernd Eckenfels e...@zusammenkunft.net



On 7/25/15 8:19 AM, Bernd Eckenfels wrote:
 Am Sat, 25 Jul 2015 07:57:57 -0700
 schrieb Phil Steitz phil.ste...@gmail.com:
 BasicDataSource allows you to directly set the abandoned connection
 log writer.  Your code uses PoolingDataSource, which requires you to
 configure the underlying object pool manually, so you need to use
 the setLogWriter method of the AbandonedConfig instance you use to
 configure the pool.
 I tried that as well, but it did not help.

 Try looking at a thread dump.  Another possibility is that the
 evictor thread is hung waiting on a driver action.  
 Thats why the output contains a thread dump, there the evictor thread
 was idle.

 I guess I would have to debug it.

I am sorry I did not see the thread dump.  I did some testing and I
think I now know what is going on.  If you add
acfg.setLogWriter(new PrintWriter(System.out, true));

you will see the stack trace.  This is a bug in Commons Pool.  The
default logwriter is not set up to autoflush and the
DefaultPooledObject (what DBCP uses) does not flush the
PrintWriter.  Thanks for reporting it and sorry to be slow to dig
into it.

I will open the pool bug.

Phil

 Gruss
 Bernd






-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] Abandon Connection - not Logging in standalone

[Phil Steitz]

On 7/23/15 4:18 PM, Bernd Eckenfels wrote:
 Hello,

 I noticed that with Oracle drivers you cannot kill a busy connection
 by closing it (since close(), _getPC(), isOpen() and stmt.close() all
 will synchronize on the (busy) physical connection.

 I noticed this in a custom persistence layer, so I thought to check out
 if DBCP handles this situation.

DBCP does nothing special itself. It relies on JDBC drivers to open
and close connections.

 I have a test program here:

 https://gist.github.com/ecki/345ee08ac97820972fe7

 First of all it does not log DBCP messages. I am not sure what is
 needed to get it going. As you can see in the output the deliberate
 debug() message is printed so the general LogFactory should be set up.

DBCP itself generates very few log messages.   You should be getting
those.  See below, though, for what you are looking for.

 When I run the program with acfg.setRemoveAbandonedTimeout(12) it
 suceeds, since the c3 is only idle for 10s. When I run it with (4) it
 will (so it seems) close the connection and I get a exception on next
 use. However I dont see a log message about the abandoned connection
 and the usage tracking. Ideas?

Do you see the abandoned object stack trace on the console
(System.out)?  The abandoned trace is not logged by DBCP.  It is
Commons Pool that generates the abandoned trace and writes it to the
Printwriter configured via its setLogWriter method.  The default is
System.out.

 A second thing, from the abandoned tracker I had the impression the
 usage is counted at the begin of exec.

Correct.

  So if a execution takes 20s as
 in my case, it should be considered abandoned. But it seems not to be
 the case. Is it recognizing it is busy when the exec does not return?

No, it will try to close the connection when the idle object evictor
runs.  If the close raises an exception, you should see a stack
trace on System.err.  You don't see that?

Phil

 If this is the case, is there also a way to kill/destroy connections
 beeing used too long?

 Gruss
 Bernd

 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org





-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] Handling many different user accounts over time

[Phil Steitz]

On 6/18/15 8:17 AM, Oksana wrote:
 Cameron, Scott scott.cameron at sap.com writes:


 Interesting that this has not come up before.  You are right that 
 PerUserPoolDataSource does not bound the
 number of pools that can be created and SharedPoolDataSource does not 
 bound the total combined size of the
 keyed pool it maintains.  Please do open a JIRA requesting the 
 capability.
 An ugly but possibly adequate workaround would be to subclass either 
 one of the datasources above and
 override getPooledConnectionAndInfo to enforce a global bound that you 
 maintain in a new instance
 variable before calling the superclass method.

 Thanks for the confirmation of the issue.  I've entered it into JIRA 
 as DBCP-373.
 The ugly work-around you suggest is actually exactly what we ended up 
 doing to 
 get access the inner object pool object.  We're hoping we can replace 
 it with
 something nicer in the future but for now it does the trick with only 
 a few
 (admittedly stinky) lines of code.

 Cheers,
 scott

 Hi,

 I was wondering if this issue was resolved. I can see that the JIRA 
 request has been closed (https://issues.apache.org/jira/browse/DBCP-
 373). However, I have looked into the source code changes 
 (https://fisheye6.atlassian.com/changelog/commons?cs=1568055) and API 
 documentation (version 2.0) and I don't see new attributes which would 
 control the total number of user pools allocated by 
 PerUserPoolDataSource. I am seeing a bunch of new attributes added to 
 control the functionalities of individual pools, but I don't see any new 
 options to control the total number of pools.

Sorry for the slow response.  You are correct that, as stated in the
comment closing the original ticket, there is no direct way to limit
the total number of pools.  I can't think of a straightforward way
to add that feature directly without significant performance impact,
but that doesn't mean there is no way to do it.  Feel free to either
reopen the ticket or create a new one.  Patches welcome, of course!

Phil

 Thanks,
 Oksana





 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [math]plot distribution

[Phil Steitz]

On 6/19/15 2:22 AM, narjes saraie wrote:
 Hi Phil.


 I open http://s.apache.org/HTo  http://s.apache.org/HTo
 but i couldent use it.Is it possible for you to attache one  source of the
 plot?

Click on RealDistributionComparison.java and you will see the
source code.

Phil
 I dont know witch line help me to draw the plots in the user
 guide section on probability distributions?
 Im sorry that Im  very very beginner.


 thanks a lot .


 http://s.apache.org/HTo

 On Thu, Jun 18, 2015 at 8:06 AM, Phil Steitz phil.ste...@gmail.com wrote:

 On 6/18/15 6:31 AM, narjes saraie wrote:
 Hi All.
 I am beginner in java and have some data.I want to guess a distribution
 for
 my data then calculate goodness of fit (gof).
 I find distribution commons math and use it ,if i want plot my
 distribution
 or CDF or probability (Xx), how do it?
 is it any example for distribution and plotting it.
 We don't have graphical exploratory data analysis tools in Commons
 Math.  You would do best to start with a package that supports
 generation of histograms if you have no idea what distribution might
 fit the data.  If you do have an idea of what the distribution is,
 depending on what it is, you may be able to estimate the parameters
 of the distribution from the data.  For example, if you think that
 the data are normally distributed, you can use SummaryStatistics in
 the stat package to get the standard statistical estimators of the
 mean and standard deviation, which together determine a normal
 distribution. Then you could do a Kolmogorov-Smirnov test (or group
 the data and use a G- or ChiSquare test) to assess goodness of fit.
 Other distributions have different parameters and different
 statistical estimators.

 We don't currently support plotting distributions in Commons Math,
 but the user guide code contains an example showing how to do it
 with some external tools.  Have a look at the plots in the user
 guide section on probability distributions [1] and the source for
 the code that generates those plots [2].

 Phil

 [1]
 http://commons.apache.org/proper/commons-math/userguide/distribution.html
 [2] http://s.apache.org/HTo

 thanks.



 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [math]plot distribution

[Phil Steitz]

On 6/19/15 5:34 AM, narjes saraie wrote:
 Hi Gilles.

 I do it and  Thank you for your patient guidance.

Look at the link I posted to the sources used to generate the user
guide.  You can see plots in the user guide itself.  Those plots
were generated with freely available OSS tools.

Phil

 cheers


 On Fri, Jun 19, 2015 at 2:51 AM, Gilles gil...@harfang.homelinux.org
 wrote:

 Hello.

 On Fri, 19 Jun 2015 01:46:25 -0700, narjes saraie wrote:

 Hi Gilles,


 I run this example in Netbeans  and struct t distribution

 TDistribution t = new TDistribution(29);
 double lowerTail = t.cumulativeProbability(-2.656); // P(T(29) =
 -2.656)
 double upperTail = 1.0 - t.cumulativeProbability(2.75);// P(T(29) = 2.75)
 double m = t.density(3);
 out.println( m );


 I want to see how I can plot this distribution or
 t.cumulativeProbability(Xx)?

 As Phil already told you, there is no plotting utility within Commons Math.
 Commons Math is only a library of mathematical algorithms.

 In your above case, you could write the values to an ASCII file (and then
 use a plotting tool to display them).
 In java, this would be a code similar to the following (import and
 exception
 handling omitted):

 ---CUT---
 String filename = distribution.dat;
 BufferedOutputStream out = new BufferedOutputStream(new
 FileOutputStream(filename));

 TDistribution t = new TDistribution(29);

 final double min = -10;
 final double max = 15;
 final double delta = 0.01;
 for (double x = min; x  max; x += delta) {
 final double upperTail = 1 - t.cumulativeProbability(x);
 out.println(x +   + upperTail);
 }

 out.close();
 ---CUT---

 One plotting tool is gnuplot:
   http://gnuplot.sourceforge.net/demo/


 HTH,
 Gilles



 thanks .

 On Thu, Jun 18, 2015 at 7:38 AM, Gilles gil...@harfang.homelinux.org
 wrote:

  Hi.

 On Thu, 18 Jun 2015 06:31:00 -0700, narjes saraie wrote:

  Hi All.
 I am beginner in java and have some data.I want to guess a distribution
 for
 my data then calculate goodness of fit (gof).
 I find distribution commons math and use it ,if i want plot my
 distribution
 or CDF or probability (Xx), how do it?
 is it any example for distribution and plotting it.
 thanks.


 Perhaps it would be clearer to see what you mean with a code example.

 If you want to plot a distribution implemented in Commons Math, you'd
 call one the methods defined in the RealDistribution interface
 (e.g. density(double x)).

 But maybe that the functionality which you are looking for is defined
 in package org.apache.commons.math3.stat.inference.

 If you need to fit data to a model (e.g. find the parameters of an
 assumed distribution), it is perhaps the least-squares fitting which
 you are looking for.  For a univariate function, there is a utility
 implemented as org.apache.commons.math3.fitting.SimpleCurveFitter.


 HTH,
 Gilles



 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [math]plot distribution

[Phil Steitz]

On 6/18/15 6:31 AM, narjes saraie wrote:
 Hi All.
 I am beginner in java and have some data.I want to guess a distribution for
 my data then calculate goodness of fit (gof).
 I find distribution commons math and use it ,if i want plot my distribution
 or CDF or probability (Xx), how do it?
 is it any example for distribution and plotting it.

We don't have graphical exploratory data analysis tools in Commons
Math.  You would do best to start with a package that supports
generation of histograms if you have no idea what distribution might
fit the data.  If you do have an idea of what the distribution is,
depending on what it is, you may be able to estimate the parameters
of the distribution from the data.  For example, if you think that
the data are normally distributed, you can use SummaryStatistics in
the stat package to get the standard statistical estimators of the
mean and standard deviation, which together determine a normal
distribution. Then you could do a Kolmogorov-Smirnov test (or group
the data and use a G- or ChiSquare test) to assess goodness of fit. 
Other distributions have different parameters and different
statistical estimators.

We don't currently support plotting distributions in Commons Math,
but the user guide code contains an example showing how to do it
with some external tools.  Have a look at the plots in the user
guide section on probability distributions [1] and the source for
the code that generates those plots [2].

Phil

[1]
http://commons.apache.org/proper/commons-math/userguide/distribution.html
[2] http://s.apache.org/HTo

 thanks.




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

[ANNOUNCEMENT] Apache Commons Pool 2.4.1 released

[Phil Steitz]

The Apache Commons Team is pleased to announce the release of Apache Commons 
Pool 2.4.1.

The Apache Commons Pool open source software library provides an object-pooling 
API and a number of object pool implementations.

No client code changes are required to migrate from any of the 2.x versions to 
2.4.1. Users of version 1.x should consult the migration guide on the Commons 
Pool web site.

Source and binary distributions are available for download from the Apache 
Commons download site:
http://commons.apache.org/proper/commons-pool/download_pool.cgi

When downloading, please verify signatures using the KEYS file available at the 
above location.

Full details of all the changes in 2.4.1 can be found in the changelog:
http://commons.apache.org/proper/commons-pool/changes-report.html

For complete information on Commons Pool, including instructions on how to 
submit bug reports, patches, or suggestions for improvement, see the Apache 
Commons Pool website:
http://commons.apache.org/proper/commons-pool/

Phil Steitz, on behalf of the Apache Commons community



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] No operations allowed after statement closed

[Phil Steitz]

On 5/27/15 3:10 PM, Ajay kumar Gour wrote:
 Hi there,



 One of the application I am working on is using commons-dbcp-1.2.2.jar.
 Below is my data source configuration –



 Resource id=myDS type=javax.sql.DataSource

   JdbcDriver com.mysql.jdbc.Driver

   JdbcUrl jdbc:mysql://localhost:3306/test?relaxAutoCommit=true

   UserName root

   Password 

   JtaManaged true

   MaxWait 0

   PoolPreparedStatements true

   MaxOpenPreparedStatements 20

   MaxActive 5

   MinEvictableIdleTime 5 minutes

   TimeBetweenEvictionRuns 12 millisecond

   NumTestsPerEvictionRun 5

   TestWhileIdle false

   TestOnBorrow true

   ValidationQuery Select 1 from dual

 /Resource



 Problem is, ever since I enabled eviction (also reduced
 MaxOpenPreparedStatement value), I am continuously getting following error



 Caused by:
 com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
 operations allowed after statement closed.

 at
 sun.reflect.GeneratedConstructorAccessor421.newInstance(Unknown Source)

 at
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

 at
 java.lang.reflect.Constructor.newInstance(Constructor.java:513)

 at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)

 at com.mysql.jdbc.Util.getInstance(Util.java:386)

 at
 com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1014)

 at
 com.mysql.jdbc.SQLError.createSQLException(SQLError.java:988)

 at
 com.mysql.jdbc.SQLError.createSQLException(SQLError.java:974)

 at
 com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)

 at
 com.mysql.jdbc.StatementImpl.checkClosed(StatementImpl.java:463)

 at
 com.mysql.jdbc.PreparedStatement.setInternal(PreparedStatement.java:3819)

 at
 com.mysql.jdbc.PreparedStatement.setLong(PreparedStatement.java:3849)

 at
 org.apache.commons.dbcp.DelegatingPreparedStatement.setLong(DelegatingPreparedStatement.java:123)

 at
 org.apache.commons.dbcp.DelegatingPreparedStatement.setLong(DelegatingPreparedStatement.java:123)

 at
 org.apache.commons.dbcp.DelegatingPreparedStatement.setLong(DelegatingPreparedStatement.java:123)

 at org.hibernate.type.LongType.set(LongType.java:42)



 I have tried updating dbcp-1.2.2 to dbcp-1.4, but same error. Could
 somebody please help me understand why is this happening? Is there a bug in
 DBCP’s statement pooling? It seems that statements are getting closed but
 they are not removed from the pool.

If you can separate the evictor activation change from the
maxOpenPreparedStatement change that will help narrow down the
problem.  One possibility is that your code is closing statements,
but holding references to them and then reusing them.  Check that
that is not happening.  What could be going on is with large
maxOpenPreparedStatement, code closes a statement, but holds a
reference to it.  Closing it returns it to the statement pool on the
connection, but does not close the physical connection.  When you
impose a maxOpenPreparedStatement limit, some of those returning
statments are going to get closed on return to the pool.  If your
client code is holding references to them and trying to do things
with them, the driver exceptions below would make sense.

Phil



 Thanks,

 -Ajay




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [configuration] Apache Commons Configuration Version 2

[Phil Steitz]

On 5/10/15 9:10 AM, James Carman wrote:
 As do people who try to blow dry their hair while they are in the bathtub,
 but they don't stop making hairdryers. They put a sticker on the cord which
 tells people not to do it.

The problem is jar hell resulting from o.a.c.foox getting out in
the wild before the x API is stable.  For Commons components, that's
more like letting someone bring a hairdryer to a Roman bath. 

Phil
 On Sun, May 10, 2015 at 10:25 AM sebb seb...@gmail.com wrote:

 On 9 May 2015 at 21:07, Ralph Goers ralph.go...@dslextreme.com wrote:
 You can publish an alpha or a beta and still change the API. That really
 isn’t locked in until the first official GA release.

 In theory, perhaps, at least with an alpha release.

 However how many people take note of the alpha/beta status of the code?

 Any project that depends on an alpha/beta release ought itself to be
 labelled as alpha/beta - does that happen?
 If not, then downstream projects may get a nasty shock.

 Ralph

 On May 7, 2015, at 3:44 AM, sebb seb...@gmail.com wrote:

 Yes, it might cause a problem to publish an RC.

 Any publication to Maven Central effectively freezes the current public
 API.
 If a problem is then found, generally the only solution that avoids
 jar hell is to release a new version with different package name and
 different Maven coords - i.e. effectively a different product.
 This will not be a drop-in replacement, and will require source code
 changes by anyone using it.

 On 7 May 2015 at 03:54, DiFrango, Ronald (CONT)
 ronald.difra...@capitalone.com wrote:
 Yeah, I would prefer to avoid the snap-shot repo, so if nothing else
 I'll build a version locally and upload it to our local Nexus.
 One question, if this is close, would it hurt to pulish an RC into
 Maven Central?
 -Original Message-
 From: sebb [mailto:seb...@gmail.com]
 Sent: Wednesday, May 06, 2015 9:20 PM
 To: Commons Users List
 Subject: Re: [configuration] Apache Commons Configuration Version 2

 You should also be able to get it from the ASF snapshot repo.

 However note that such artifacts may change or disappear at any time,
 so should only be used for testing.
 On 6 May 2015 at 20:07, Oliver Heger oliver.he...@oliver-heger.de
 wrote:
 Hi Ron,

 Am 06.05.2015 um 03:38 schrieb DiFrango, Ronald (CONT):
 Oliver,

 I guess I could build it from source and store it in our local Nexus
 until a release version is built.
 this is of course possible. I do not expect major or breaking changes
 until the final release.

 Thanks for your feedback, this is indeed motivating :-)

 Oliver

 Ron DiFrango

 -Original Message-
 From: DiFrango, Ronald (CONT) [mailto:ronald.difra...@capitalone.com
 ]
 Sent: Tuesday, May 05, 2015 4:23 PM
 To: Commons Users List
 Subject: RE: [configuration] Apache Commons Configuration Version 2

 Oliver,

 I figured it would be something like that and that's understandable
 as is the time-table.  We definitely like the feature set of 2.x as it
 supports many of the uses cases more naturally than 1.x so would like to
 move forward with it.
 But I can't unless it's in maven central.

 Thanks,

 Ron


 -Original Message-
 From: Oliver Heger [mailto:oliver.he...@oliver-heger.de]
 Sent: Tuesday, May 05, 2015 4:20 PM
 To: Commons Users List
 Subject: Re: [configuration] Apache Commons Configuration Version 2

 Hi Ron,

 Am 05.05.2015 um 22:09 schrieb DiFrango, Ronald (CONT):
 All,

 Is Apache commons configuration in maven central or some other
 repo?  I'm using Gradle and pointing to an internal Nexus repo that is a
 proxy for Maven central and I can't resolve it as follows:
 compile group: 'org.apache.commons', name:
 'commons-configuration2', version: '2.0-alpha2'

 because it is only an alpha release, this version has not been
 deployed to a public Maven repository. We did this by intension to avoid
 potential jar hell if there are still binary incompatible changes before
 the final release.
 I am working to get the final version 2.0 out, but unfortunately my
 time is currently pretty limited. So I cannot predict a release schedule.
 Oliver

 Thanks,

 Ron

 

 The information contained in this e-mail is confidential and/or
 proprietary to Capital One and/or its affiliates. The information
 transmitted herewith is intended only for use by the individual or entity
 to which it is addressed.  If the reader of this message is not the
 intended recipient, you are hereby notified that any review,
 retransmission, dissemination, distribution, copying or other use of, or
 taking of any action in reliance upon this information is strictly
 prohibited. If you have received this communication in error, please
 contact the sender and delete the material from your computer.
 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org

 

Re: [DBCP2] Timeout on PoolingDataSource#getConnection() ?

[Phil Steitz]

On 4/30/15 1:40 PM, Shawn Heisey wrote:
 I'm using dbcp2-2.1 and pool2-2.3.

 Is there any kind of timeout configurable (or even needed) when calling
 getConnection() on a PoolingDataSource?  I was looking over my code for
 possible problems and realized that I have no idea whether this call
 might block indefinitely or whether it will always finish (or throw an
 exception) within some reasonable timeframe.  The javadocs don't offer
 anything useful, which hopefully means that there is no possibility of a
 significant or indefinite pause.

 Below is the code I'm using to initialize.  In this code, dsMaster is an
 instance of javax.sql.DataSource, and one of the objects where I call
 getConnection().

  ConnectionFactory cfMaster = new
 DriverManagerConnectionFactory(masterUrl, dbUser, dbPass);
  PoolableConnectionFactory pcfMaster = new
 PoolableConnectionFactory(cfMaster, null);
  pcfMaster.setValidationQuery(validationQuery);
  pcfMaster.setValidationQueryTimeout(5);
  opMaster = new GenericObjectPool(pcfMaster);
  pcfMaster.setPool(opMaster);
  dsMaster = new PoolingDataSource(opMaster);

Client blocking / timeout above is controlled by the properties set
on the GenericObjectPool.  By default, the blockWhenExhausted
property of a GOP is true and maxWaitMillis is -1, which means
clients will block indefinitely.   If you want clients to fail when
there are no connections available, set blockWhenExhausted to
false.  If you want to block for n millis, leave blockWhenExhausted
true and set maxWaitMillis to n.   Alternatively, you could just use
BasicDataSource, which is a little simpler to set up and exposes the
pool properties directly.

Phil

 Thanks,
 Shawn


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org





-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: POOL - Question about different time out for borrow and add object

[Phil Steitz]

On 4/29/15 8:59 AM, vijay wrote:
 Hi ,

 I want to know if i can achieve this. Is there any way to specify time out
 for the addobject and borrow object methods?

 What i am really looking is when i start the pool i am prefilling it with
 add object for this i can wait for more time to geth the pool initialized.
 The backned system i have takes more time to crate connections

 But at the time of borrow i cannot have the client wait for long time if no
 idle connection is available and the factory make object method might take
 more time to  return object.

 So is there anyway i can tall factory method to wait for x time for add
 obkject and y time for borrow object calls

There is no timeout for addObject.  It is just a no-op if there is
no capacity to add to the pool when you call it.  If you are
prefilling at startup, that should not be a problem.  If you
actually *want* a timeout for the adds, then you need to implement
it in your factory itself.

Phil



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: POOL - Question about different time out for borrow and add object

[Phil Steitz]

On 4/29/15 12:55 PM, vijay wrote:
 Hi Phil..

 Thanks for response. i agree that add object is not problem as i am pre
 filling the pool and i can wait until the pool gets filled. But the real
 prob is with borrow object where i have to call the external system if
 there are no idle objects and the pool didn't reach the max size.

 In this case the pool will try to create new object by using factory make
 object method. The same method which add object also uses to add object to
 pool. I this case the borrow objeect call will get blocked until new
 connection is provided by external system (This could be some time more
 than 5 mins)

 The factory instance of the pool at this particular point has no
 information of whether the call has happened from borrow object or add
 object of pool to handle the time out based on add or borrow...This is
 where iam stuck ...

 can you help me how to implement it ..

Sorry, but pool can't really help with this use case.  The only
suggestion that I can make is to have your factory maintain state
itself - i.e., have it expose a property, say timeLimited and a
timeout value.  Have timeLimited false when you are preloading and
then set it to true and have the factory enforce the timeout. 
Another thing to note is that pool's borrowObject timeout limits the
amount of time that a thread waits for either an idle object or for
capacity to create one.  If a thread does get to create an object,
it will wait until the makeObject completes - i.e. (at least as of
2.3) the pool will not interrupt a thread waiting for a factory
method to complete.  That means that for your purpose, the
borrowObject timeout will not work either.

So basically the only solution that I can offer you is to have the
factory itself control the timeout.

Phil

 Thanks

 On Wed, Apr 29, 2015 at 2:19 PM, Phil Steitz phil.ste...@gmail.com wrote:

 On 4/29/15 8:59 AM, vijay wrote:
 Hi ,

 I want to know if i can achieve this. Is there any way to specify time
 out
 for the addobject and borrow object methods?

 What i am really looking is when i start the pool i am prefilling it with
 add object for this i can wait for more time to geth the pool
 initialized.
 The backned system i have takes more time to crate connections

 But at the time of borrow i cannot have the client wait for long time if
 no
 idle connection is available and the factory make object method might
 take
 more time to  return object.

 So is there anyway i can tall factory method to wait for x time for add
 obkject and y time for borrow object calls
 There is no timeout for addObject.  It is just a no-op if there is
 no capacity to add to the pool when you call it.  If you are
 prefilling at startup, that should not be a problem.  If you
 actually *want* a timeout for the adds, then you need to implement
 it in your factory itself.

 Phil

 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org





-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp]Closed Connection or Exhausted Resultset error

[Phil Steitz]

On 4/27/15 11:26 PM, Rohini T Nagaraj wrote:
 Hi,
 We are using the DBCP -org.apache.commons-dbcp-1.2.2.jar for connection 
 pooling.

I don't think so.  You are using RS.isClosed below, which is JDK
1.6+.  So assume you are using DBCP 1.4?  Pls verify  DBCP and pool
version.
  Connection pool is created as shown in below code and had set the 
 connection pool setting  (removed evictable time settings). There is no 
 issue in getting a connection. This part of the code works fine.

   GenericObjectPool connectionPool = new GenericObjectPool();
   connectionPool.setMaxActive(25); 
 connectionPool.setMaxIdle(25); 
 connectionPool.setTestOnReturn(false);
 connectionPool.setTestOnBorrow(true);
 connectionPool.setTestWhileIdle(false);
 GenericKeyedObjectPoolFactory statementPoolFactory = new 
 GenericKeyedObjectPoolFactory(null
 ,-1,GenericKeyedObjectPool.WHEN_EXHAUSTED_FAIL,0,1,dbEndPt.getMaxPreparedStatement());
 ConnectionFactory connectionFactory = new 
 DataSourceConnectionFactory(ds);
 boolean defaultReadOnly = false;
 CIPoolableConnectionFactory poolableConnectionFactory = new  
 CIPoolableConnectionFactory(connectionFactory,connectionPool,statementPoolFactory,getValidationQuery(dbEndPt),defaultReadOnly,
 false); 
 PoolingDataSource pds = new 
 DelegatePoolingDataSource(connectionPool);
 pds.setAccessToUnderlyingConnectionAllowed(true);
   pds.getConnection();


 Later when we run this part of the code , it takes almost 12mins to get 
 the resultset. Since customer has 37k packages ,it might have taken this 
 time. But after that we get the error as Closed Resultset and it does not 
 enter the while loop.This happens only when we use datasource connection 
 pooling. Otherwise this part of the code works fine in a standalone 
 program and takes 10mins to get the resultset.Please suggest us connection 
 pool settings to handle the 37k packages on the Oracle side.Thanks.

 ResultSet rs = metaData.getProcedureColumns(package_name, 
 schema, storedProcedure, null ); 
  
  
   logger.warning(Got resultset.Is ResultSet closed = 
 +rs.isClosed()+\n);

   if (rs != null){

  
   logger.warning(rs is not equal to NULL.Is ResultSet 
 closed = +rs.isClosed()+\n);

   while( rs.next() ) {

 //code to get the params
 }

 ERROR:
 Apr 27, 2015 9:17:36 AM com.approuter.framework.util.ResourceBundleHelper 
 getString
 WARNING: Could not find the resource menu.edit.label in bundle: 
 orchestration using the resource bundle service. Fallback to old style

 Apr 27, 2015 9:29:07 AM 
 com.approuter.studio.connectors.database.util.DBHelper 
 getStoredProcedureMetaData
 WARNING: Got resultset.Is ResultSet closed = false

 Apr 27, 2015 9:29:07 AM 
 com.approuter.studio.connectors.database.util.DBHelper 
 getStoredProcedureMetaData
 WARNING: rs is not equal to NULL.Is ResultSet closed = true

 java.sql.SQLException: Closed Resultset: next
  at 
 oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:565)
  at 
 com.approuter.studio.connectors.database.util.DBHelper.getStoredProcedureMetaData(DBHelper.java:1052)//this
  
 line points to the whileloop

I am having a hard time piecing this together.  Pls include omitted
code.  Does the code call rs.next within the body of the loop as well?

Phil


 Thanks and Regards,
 Rohini T Nagaraj,
 WebSphere CastIron QA Team,
  IBM INDIA PRIVATE LIMITED,
 DC1-3A-003,DLF IT PARK,Chennai - 600089
 Extn # : 21820 and Mobile #: 9962020675



 From:   Phil Steitz phil.ste...@gmail.com
 To: Commons Users List user@commons.apache.org
 Date:   04/22/2015 07:03 PM
 Subject:Re: [dbcp]Closed Connection or Exhausted Resultset error



 On 4/22/15 5:18 AM, Rohini T Nagaraj wrote:
 Hi All,

 The Oracle DB has 37k packages. The issue is seen after running the 
 below 
 query from the program or when using 
 DatabaseMetadata.getProcedureColumns()  in a program . We are seeing 
 these 
 errors  Exhausted Resultset (SQL Code =9 and vendor code=17,011)  
 or 
 Closed Connection :next (SQL code=08003 , Vendor code=17,008).
 Please post a stack trace showing one or both of these errors.
 Something happening to connection or resultset getting closed after 
 running the query/getProcedureColumns call. We have investigated further 
 and see this issue of Exhausted Resultset/Connection getting closed only 
 when we use the datasource connection pool logic.
 When we run a standalone java program with the query/getProcedureColumns 
 call , it took 10mins for the customer to get the output and there were 
 no 
 issues.We want the same behavior with datasource connection pool logic .


 We are using Tomcat sevrer and 
 org.apache.commons.pool.impl.GenericObjectPool connection pool.We have 
 these connection pool settings done

Re: [dbcp]Closed Connection or Exhausted Resultset error

[Phil Steitz]

On 4/28/15 9:47 AM, Rohini T Nagaraj wrote:
 Hi ,

 We use org.apache.commons-dbcp-1.2.2.jar and 
 com.springsource.org.apache.commons.pool-1.4.0.jar. Looks like we should 
 be using a java less than 1.4 as per this link 
 http://commons.apache.org/proper/commons-dbcp/. But we are using java 1.7. 
 Please confirm if this incompatibility is causing this issue. Here is the 
 full method for review.Thanks.

That would be a problem.  Given that your code compiles, I suspect
what is going on is that you are somehow accessing connections
directly (not actually using DBCP), which will cause all kinds of
problems.  You need to both upgrade dbcp and pool versions and use
DBCP to actually manage connections, resultsets, etc.

Phil

 public static java.util.ListStoredProcedureTableData 
 getStoredProcedureMetaData(String package_name,String schema, String 
 storedProcedure, String encoding) {
 DatabaseMetaData metaData = 
 DBConnectionManager.getDatabaseMetaData();
 java.util.ListStoredProcedureTableData list = new 
 ArrayListStoredProcedureTableData();
 String parameterName;
 String dataType;
 String type;
 String size; 
 short shortType;
 StoredProcedureTableData data;
 ResultSet rs = null;
  
 try {
  storedProcedure=storedProcedure.substring(package_name.length()+1);
   rs = metaData.getProcedureColumns(package_name, schema, 
 storedProcedure, null ); //$NON-NLS-1$ 
   logger.warning(Got resultset.Is ResultSet closed = 
 +rs.isClosed()+\n);
   if (rs != null){ 
   logger.warning(rs is not equal to NULL.Is ResultSet 
 closed = +rs.isClosed()+\n);
   while( rs.next() ) { 
   logger.warning(In While loop.Is ResultSet closed = 
 +rs.isClosed()+\n);
 parameterName = rs.getString( 4 ); // COLUMN_NAME (7) for 
 query 
 parameterName = getValidParameterName( parameterName );
 logger.warning(The Parametername = +parameterName);
 logger.warning(After 1st param.Is ResultSet closed = 
 +rs.isClosed()+\n); 
 dataType = getDataType( rs.getInt( 6 ), rs.getString( 7 ) 
 ); //DATA_TYPE, TYPE_NAME 
 shortType = rs.getShort( 5 );//COLUMN_TYPE
 type = getColumnType( shortType );
 logger.warning(The DataType = +dataType);  
 logger.warning(The Type = +type+ TYPE_RETURN = 
 +TYPE_RETURN);
 size = Integer.toString( rs.getInt( 8 ) ); 
 logger.warning(The Size = +size); 
 short scale = rs.getShort( 10 ); //(18) for query
 logger.warning(The Scale = +scale);
 if( scale  0 ) {
   size = size + , + scale; //$NON-NLS-1$
 } 
 data = new StoredProcedureTableData( true, parameterName, 
 dataType, type, encoding, size , TYPE_RETURN.equals(type));
 list.add( data );
   }
   rs.close();
   rs = null;
 } 
 }catch( SQLException e ) {
   connectionError( e );
   e.printStackTrace();
 }finally { 
 if (rs != null) {
   try { 
   rs.close(); 
   logger.warning(In finally.Is ResultSet closed = 
 +rs.isClosed()+\n);
   } catch (SQLException e) {
   e.printStackTrace();
   }
 }
 rs = null;
 }
 return list;
 }



 Thanks and Regards,
 Rohini T Nagaraj,
 WebSphere CastIron QA Team,
  IBM INDIA PRIVATE LIMITED,
 DC1-3A-003,DLF IT PARK,Chennai - 600089
 Extn # : 21820 and Mobile #: 9962020675



 From:   Phil Steitz phil.ste...@gmail.com
 To: Commons Users List user@commons.apache.org
 Date:   04/28/2015 07:14 PM
 Subject:Re: [dbcp]Closed Connection or Exhausted Resultset error



 On 4/27/15 11:26 PM, Rohini T Nagaraj wrote:
 Hi,
 We are using the DBCP -org.apache.commons-dbcp-1.2.2.jar for connection 
 pooling.
 I don't think so.  You are using RS.isClosed below, which is JDK
 1.6+.  So assume you are using DBCP 1.4?  Pls verify  DBCP and pool
 version.
  Connection pool is created as shown in below code and had set the 
 connection pool setting  (removed evictable time settings). There is no 
 issue in getting a connection. This part of the code works fine.

   GenericObjectPool connectionPool = new GenericObjectPool();
   connectionPool.setMaxActive(25); 
 connectionPool.setMaxIdle(25); 
 connectionPool.setTestOnReturn(false);
 connectionPool.setTestOnBorrow(true);
 connectionPool.setTestWhileIdle(false);
 GenericKeyedObjectPoolFactory statementPoolFactory = new 
 GenericKeyedObjectPoolFactory(null

 ,-1

Re: [dbcp]Closed Connection or Exhausted Resultset error

[Phil Steitz]

On 4/22/15 5:18 AM, Rohini T Nagaraj wrote:
 Hi All,

 The Oracle DB has 37k packages. The issue is seen after running the below 
 query from the program or when using 
 DatabaseMetadata.getProcedureColumns()  in a program . We are seeing these 
 errors  Exhausted Resultset (SQL Code =9 and vendor code=17,011)  or 
 Closed Connection :next (SQL code=08003 , Vendor code=17,008).

Please post a stack trace showing one or both of these errors.

 Something happening to connection or resultset getting closed after 
 running the query/getProcedureColumns call. We have investigated further 
 and see this issue of Exhausted Resultset/Connection getting closed only 
 when we use the datasource connection pool logic.
 When we run a standalone java program with the query/getProcedureColumns 
 call , it took 10mins for the customer to get the output and there were no 
 issues.We want the same behavior with datasource connection pool logic .

  
 We are using Tomcat sevrer and 
 org.apache.commons.pool.impl.GenericObjectPool connection pool.We have 
 these connection pool settings done. 

Are you using DBCP or are you just trying to pool the connections
manually using Commons Pool?  What version of pool and / or DBCP? 
How exactly are you creating the connection pool?

 connectionPool.setMaxActive(25);
 connectionPool.setMaxIdle(1);
 connectionPool.setMinEvictableIdleTimeMillis(12);
 connectionPool.setTimeBetweenEvictionRunsMillis(1000);
 connectionPool.setTestOnReturn(false);
 connectionPool.setTestOnBorrow(true);
 connectionPool.setTestWhileIdle(false); 

Why do you have maxIdle set to 1? That is going to cause connections
to get closed when they are returned whenever there is one idle
connection already in the pool.  This will effectively defeat the
purpose of the pool.  Also, with such rigid control on idle
connections, why do you also have the evictor configured to run? 
Every second?  Unless you have special reasons for these settings, I
would change maxIdle to 25 and drop the minEvictableIdleTimeMillis
and timeBetweenEvictionRunsMillis (default will be to have no
evictor runs).

 Hence please let us know if there is any fine tuning to be done to make 
 query or DatabaseMetadata.getProcedureColumns() to work when we use 
 datasource connection pool logic.Please suggest us any other timeout or if 
 there is anything else which can help us in this scenario.Thanks
  

 SQL QUERY :
 SELECT * FROM SYS.ALL_ARGUMENTS WHERE PACKAGE_NAME = 'PKG_TEST' AND 
 OBJECT_NAME = 'PRC_INSERT' AND OWNER = 'APPS' ORDER BY SEQUENCE

 ERROR:
 java.sql.SQLRecoverableException: Closed Connection: next
 at 
 oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:233)



 Thanks and Regards,
 Rohini T Nagaraj,
 WebSphere CastIron QA Team,
  IBM INDIA PRIVATE LIMITED,
 DC1-3A-003,DLF IT PARK,Chennai - 600089
 Extn # : 21820 and Mobile #: 9962020675



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Apparent deadlock related to getConnection

[Phil Steitz]

On 4/21/15 10:43 PM, Shawn Heisey wrote:
 On 4/21/2015 6:28 PM, Phil Steitz wrote:
 Thanks for providing the full stack traces, Shawn.  I assume the
 pool config is the same that you posted before, right?  Also, the
 mysql driver seems to be taking initiative to try to clean up
 abandoned connections itself.  That is holding some locks; but I
 don't see any deadlocks in the dumps.  I will look more at this
 later tonight.  I think it is possible there is a pool bug lurking
 in here. 

Confirmed that what I was looking into was not a pool bug (actually
previously resolved bug).

Phil

  One question: are you seeing connections fail validation? 
 I think you said you were testing on borrow / return.  Is that
 right?  Are you just letting the pool destroy the bad ones or are
 you destroying them yourself?
 Thanks for your reply.

 I have since learned that this was not an actual deadlock.
   Somebody
 dropped entries into the config database which asked my program to
 reindex 56 million rows (nearly one third of the entire database) into
 Solr.  That is not the kind of thing I intended that capability to
 accomplish, because it takes WAY too long.

 My program was working fine ... but sending watchdog alarms every
 fifteen minutes because a normal cycle was taking hours longer than
 expected.  I removed the reindex entries and restarted the program
 instances, now everything is working fine again.

 Thanks,
 Shawn


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Apparent deadlock related to getConnection

[Phil Steitz]

On 4/21/15 4:47 PM, Shawn Heisey wrote:
 On 4/14/2015 6:06 AM, James Carman wrote:
 You may want to consider something like Spring's JdbcTemplate class to
 avoid a lot of this.
 I don't know anything about Spring, although I didn't know anything
 about DBCP either, before I wrote this code.  I suspect that Spring is
 much larger and more complicated, though.

 Another deadlock problem has happened.

 Two separate copies of the program that both use the same MySQL database
 appear to have locked up at the same time on different hosts, using
 different JVMs.  It has been over three hours since the first not
 updating alarms from the watchdog thread started coming in, so this is
 definitely not a temporary deadlock.  I just barely obtained these
 stacktraces:

 https://www.dropbox.com/s/yudxe1cikm0chnd/stacktrace-idxbuild-bigindy5.txt?dl=0
 https://www.dropbox.com/s/g41mldfxpttb824/stacktrace-idxbuild-idxa1.txt?dl=0

 Included in each of those files are the three bits of information
 requested from DBCP, which were automatically logged by the watchdog
 thread -- active, idle, and the pool objects.  That information is from
 the first alarm on each host, running a separate copy of the program.

 This time, I see no evidence of locks held by my own code.  All the
 locks present are in libraries that I am using.

 It wasn't immediately apparently to me, looking at the stacktraces,
 where the deadlock is.  I think this is different than the deadlock that
 started this thread.

Thanks for providing the full stack traces, Shawn.  I assume the
pool config is the same that you posted before, right?  Also, the
mysql driver seems to be taking initiative to try to clean up
abandoned connections itself.  That is holding some locks; but I
don't see any deadlocks in the dumps.  I will look more at this
later tonight.  I think it is possible there is a pool bug lurking
in here.  One question: are you seeing connections fail validation? 
I think you said you were testing on borrow / return.  Is that
right?  Are you just letting the pool destroy the bad ones or are
you destroying them yourself?

Phil

 Thanks,
 Shawn


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org





-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: Pool blocking but not hitting max size?

[Phil Steitz]

On 4/13/15 2:39 PM, Dan wrote:
 repeatedly seen this happen and would appreciate some input or advice.
 We're using version 1.6 of the commons pool, I don't believe we could
 upgrade without good reason.
 From the line numbers in the stack trace, it looks like you are
 actually running pool 1.3, which is, well, ancient.  You should
 verify the version and if it is as I suspect, you should definitely
 upgrade.  Just have a look at the change log for the many, many
 issues that have been resolved since 1.3.  That version should be
 deadlock-free, though it achieves that by extreme
 over-synchronization.  Both borrow and return are fully synchronized
 (threads are waiting on the pool monitor in the dump below).
 Version 1.3 is the least performant version of commons pool.
 My apologies, I just searched the .ear we deploy for what I thought
 was the commons pool and found commons-pool-1.6.jar, so I figured that
 was the active version. Now that I look again, there are 20 .jar files
 all containing the word commons in the file name, with varying version
 numbers. Like I said, I'm just a lowly server admin watching this app
 crash.

 Additionally, the maxIdle setting limits the number of instances
 that can be idle in the pool to just 8.  That means that when
 instances are returned and there are already 8 idle, the returning
 instances are destroyed.  When more load arrives, you then have to
 wait for them to be created, which in v 1.3 causes all threads to
 wait on the factory.  If you can afford to have more instances idle,
 you should increase that number.
 In what sense do you say 'afford'? The servers have more than enough
 CPU/RAM available when this happens. Would you expect any meaningful
 overhead from bumping this number up to 50 or 100 during normal
 operations? We have test environments, but have struggled to create
 similar loads to production in them, so it may be difficult to fully
 test this change.

 You will likely get immediate relief by increasing maxIdle to
 several hundred or even the maxActive number; but you really should
 upgrade to a more recent version.  See the pool web page for JDK
 requirements and version compatibility.
 Thanks, I was very surprised when I saw how many versions behind our
 code looked from my naive glancing. I will suggest the maxIdle change
 and go from there.

 Just to clarify, the reason these threads are (likely) blocking is
 that we have 8 idle pool members, so every single request thereafter
 will cause a synchronized construct/destruct which blocks the entire
 pool until it completes, effectively limiting throughput to one
 request at a time until load goes down. I'd just like something
 meaningful to pass onto the developers.

Let me try to explain a little better so you can make the right
decisions before and after upgrade.

The maxActive setting (renamed maxTotal in v. 2) governs the total
number of instances in circulation - checked out or idle waiting to
be checked out - at a given time.  The maxIdle setting limits the
number of instances that can sit idle in the pool.  Given your
settings, a spike in demand could cause 2048 instances to be created
and handed out to clients.  On return, any arriving back when there
were more than 8 instances idle would get destroyed.  In general
maxIdle  maxTotal under variable load will result in a lot of
object creation / destruction.  If your environment can allow
maxIdle == maxTotal (or at least not hugely less), you can avoid
this churn.  Sometimes people want to reallocate resources after
load subsides so they set maxIdle  maxActive.

Now, pool 1.3 makes the pain associated with object churn much worse
because the factory create and destroy methods are executed while
holding *global* locks on the pool.  That means all threads waiting
on borrow or return have to wait for the factory methods to complete.

Morals of the story:

1.  Upgrade to a modern version of pool (1.5.7+) where the factory
methods don't block the pool
2.  Consider setting maxIdle closer or equal to maxActive.

Phil

 I appreciate the input!

 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org





-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: Pool blocking but not hitting max size?

[Phil Steitz]

On 4/13/15 9:36 AM, Dan wrote:
 Hello, I'm not a developer on the project I'm supporting, but I have
 repeatedly seen this happen and would appreciate some input or advice.
 We're using version 1.6 of the commons pool, I don't believe we could
 upgrade without good reason.
From the line numbers in the stack trace, it looks like you are
actually running pool 1.3, which is, well, ancient.  You should
verify the version and if it is as I suspect, you should definitely
upgrade.  Just have a look at the change log for the many, many
issues that have been resolved since 1.3.  That version should be
deadlock-free, though it achieves that by extreme
over-synchronization.  Both borrow and return are fully synchronized
(threads are waiting on the pool monitor in the dump below). 
Version 1.3 is the least performant version of commons pool.

Additionally, the maxIdle setting limits the number of instances
that can be idle in the pool to just 8.  That means that when
instances are returned and there are already 8 idle, the returning
instances are destroyed.  When more load arrives, you then have to
wait for them to be created, which in v 1.3 causes all threads to
wait on the factory.  If you can afford to have more instances idle,
you should increase that number.

You will likely get immediate relief by increasing maxIdle to
several hundred or even the maxActive number; but you really should
upgrade to a more recent version.  See the pool web page for JDK
requirements and version compatibility.

Phil

 We're running WebLogic and periodically see thread count shoot up to
 the work manager maximum, and throughput grinds to a halt. All threads
 are blocked waiting on the commons pool, which I thought was strange
 as heap dumps show output like:

 Type   Name   Value
 intevictLastIndex -1
 ref_evictor   null
 int_numActive 206
 ref_factory
 org.springframework.aop.target.CommonsPoolTargetSource @ 0x610...
 ref_pool  java.util.LinkedList @ 0x610...
 long   _softMinEvictableIdleTimeMillis-1
 long   _minEvictableIdleTimeMillis180
 int_numTestsPerEvictionRun3
 long   _timeBetweenEvictionRunsMillis -1
 boolean_testWhileIdle FALSE
 boolean_testOnReturn  FALSE
 boolean_testOnBorrow  FALSE
 byte   _whenExhaustedAction   1
 long   _maxWait   -1
 int_maxActive 4096
 int_minIdle   10
 int_maxIdle   8
 booleanclosed FALSE

 So from that, I would have expected numActive to be much closer to
 4096 which is the maxActive we set. Am I looking at the wrong place?
 Why is this pool blocking? This has happened multiple times, and each
 time numActive is between 200 and 300. The pool is being used (I believe)
 to limit the number of concurrent security authorizations, so each borrow
 then pings an external service, and that external service does not appear
 overloaded at all.

 I did notice that when I drilled down into the pool's linked list, it
 only ever has 8 members in it, I would have expected numActive members
 at least. I don't fully understand the pool though.

 Here's a partial thread dump showing this behavior (addresses truncated,
 but they all blocked on the same pool object):

 [ACTIVE] ExecuteThread: '160' for queue: 'weblogic.kernel.Default
 (self-tuning)' daemon prio=10 tid=0x000... nid=0x4b1f
 waiting for monitor entry [0x000...]
java.lang.Thread.State: BLOCKED (on object monitor)
 at org.apache.commons.pool.impl.GenericObjectPool.returnObject(
 GenericObjectPool.java:916)
 - waiting to lock 0x000... (a
 org.apache.commons.pool.impl.GenericObjectPool)
 at org.springframework.aop.target.CommonsPoolTargetSource.
 releaseTarget(CommonsPoolTargetSource.java:252)
 --
 [ACTIVE] ExecuteThread: '159' for queue: 'weblogic.kernel.Default
 (self-tuning)' daemon prio=10 tid=0x... nid=0x4b1e
 waiting for monitor entry [0x000...]
java.lang.Thread.State: BLOCKED (on object monitor)
 at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(
 GenericObjectPool.java:781)
 - waiting to lock 0x000... (a
 org.apache.commons.pool.impl.GenericObjectPool)
 at org.springframework.aop.target.CommonsPoolTargetSource.getTarget(
 CommonsPoolTargetSource.java:244)
 --
 [ACTIVE] ExecuteThread: '158' for queue: 'weblogic.kernel.Default
 (self-tuning)' daemon prio=10 tid=0x000... nid=0x4b1d
 waiting for monitor entry [0x000...]
java.lang.Thread.State: BLOCKED (on object monitor)
 at org.apache.commons.pool.impl.GenericObjectPool.returnObject(
 GenericObjectPool.java:916)
 - waiting to lock 0x000... (a
 org.apache.commons.pool.impl.GenericObjectPool)
 at org.springframework.aop.target.CommonsPoolTargetSource.
 

Re: [DBCP] Apparent deadlock related to getConnection

[Phil Steitz]

On 4/12/15 2:58 AM, Shawn Heisey wrote:
 On 4/11/2015 8:48 AM, Phil Steitz wrote:
 What is going on in that thread is that it is blocked waiting for a
 connection to become available in the pool.  There is no evidence of
 deadlock there - just pool exhaustion.

 The most likely culprit is application code that is failing to close
 connections that have been borrowed from the pool.  If you don't
 close the connections on all execution paths, they will leak as
 the pool has no way of knowing that your code is finished with them
 unless and until you close them.
 I've done a review of all my database code for leaks in the past, and
 everything looked good at that time.  I can repeat that check.  The
 database portions of my code have not had any changes for quite some time.

Also try turning on removeAbandoned and logAbandoned.

 Would I see evidence of a connection leak in the show full processlist
 output at the MySQL server?  I don't see that happening.  There are only
 a handful of open connections when I check there, even when the program
 has been running for several days or weeks, running through its cycle
 once a minute.  Once all parts of the program have been run at least
 once, the connection count is stable.

Can you post your full pool config and info on versions of DBCP and
POOL?

Do you call isClosed() before closing a connection (not recommended
- that is really the issue in the ticket you commented)?

To make sure there is not an actual deadlock somewhere, you should
examine a full thread dump.  The thread you posted in the ticket is
waiting on the pool, not holding any DBCP or POOL related locks; but
it does hold this application lock:
at com.REDACTED.idxbuild.solr.Chain.doReinsert(Chain.java:1332)
- locked 0xdb0011c8 (a java.lang.Object)
Make sure no other threads in process of doing things that will
return connections to the pool aren't waiting on this lock.

Given that you are using DBCP 2.1, if you have access to a JMX
console it would also be good to look at what is happening in the
pool when your application hangs.  In particular, numActive, numIdle
and listAllObjects would be good to look at.  Looks like you are
using PoolingDataSource directly, so you need to look at the
underlying GenericObject's JMX properties.

Phil

 Thanks,
 Shawn


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org





-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP] Apparent deadlock related to getConnection

[Phil Steitz]

On 4/12/15 10:31 AM, Shawn Heisey wrote:
 On 4/12/2015 9:16 AM, Phil Steitz wrote:
 Can you post your full pool config and info on versions of DBCP and
 POOL?

 Do you call isClosed() before closing a connection (not recommended
 - that is really the issue in the ticket you commented)?

 To make sure there is not an actual deadlock somewhere, you should
 examine a full thread dump.  The thread you posted in the ticket is
 waiting on the pool, not holding any DBCP or POOL related locks; but
 it does hold this application lock:
 at com.REDACTED.idxbuild.solr.Chain.doReinsert(Chain.java:1332)
 - locked 0xdb0011c8 (a java.lang.Object)
 Make sure no other threads in process of doing things that will
 return connections to the pool aren't waiting on this lock.

 Given that you are using DBCP 2.1, if you have access to a JMX
 console it would also be good to look at what is happening in the
 pool when your application hangs.  In particular, numActive, numIdle
 and listAllObjects would be good to look at.  Looks like you are
 using PoolingDataSource directly, so you need to look at the
 underlying GenericObject's JMX properties.
 My code does not contain the string isClosed at all.

 I thought I had saved the full thread dump, but the stdout logfile was
 wiped when I restarted the program, and I didn't think to save a copy.
 I will be sure to save it next time there's a problem.

Full thread dump will be helpful.

 Here is the entire java file for my Database class, edited to redact
 the company top-level domain:

 http://apaste.info/6pX

Several methods, getSingleObject, getResultSet, closeResultSet,
doUpdate can abandon connections on exception paths.  You should
insert finally blocks in these methods to ensure that connections
are closed. 

Have a look at the PoolingDataSource example referenced [1] on the
DBCP web page for a model to use with try-catch-finally.   Not
closing connections on exception paths will cause leaks if
exceptions are thrown.  If you are seeing BuildExceptions generated
by this code, each of them is leaking a connection.

 The object that I am locking on (0xdb0011c8) is a global lock
 for my whole program which I synchronize on for a handful of
 normally-short-lived db-related operations just to be sure that they
 aren't happening simultaneously, and to help make sure that the outcome
 of certain operations is visible across threads.  I'm reasonably sure
 that there are no operations in those synchronization blocks that are
 expected to be slow, though I haven't verified that to 100% certainty.

They don't have to be slow to deadlock, if you hit the right timing
and the following kind of scenario:

Thread 1 grab the lock and hold it while waiting to get a connection
(what the thread you showed in the ticket is doing)
Threads 2, ..., n try to get the lock before returning connections
they hold

Key is to get a full thread dump and see which threads are waiting
on / holding which locks.

Phil
[1] http://s.apache.org/0Nl

 One of the operations handled in that kind of synchronization lock is
 the optimization of certain DB tables, which only happens if the size of
 those tables is below a certain threshold -- part of making sure that
 the operation completes quickly and doesn't deadlock the program.

 Here are all the jar files in my program's lib directory.  My program
 uses javamail, jrobin, dbcp, and solrj:

 commons-dbcp2-2.1.jar
 commons-io-2.4.jar
 commons-pool2-2.3.jar
 httpclient-4.4.1.jar
 httpcore-4.4.1.jar
 httpmime-4.4.1.jar
 javax.mail.jar
 jcl-over-slf4j-1.7.12.jar
 jrobin-1.5.14.jar
 jul-to-slf4j-1.7.12.jar
 log4j-1.2.17.jar
 mysql-connector-java-5.1.35-bin.jar
 noggit-0.7.jar
 slf4j-api-1.7.12.jar
 slf4j-log4j12-1.7.12.jar
 solr-solrj-5.0.0.jar
 stax2-api-3.1.1.jar
 woodstox-core-asl-4.2.0.jar
 zookeeper-3.4.6.jar

 I do not have remote JMX enabled for this application, and the server
 where it runs does not have a gui.  I will add some code that spits out
 the three requested pool stats every time the watchdog thread detects a
 problem.  I cannot predict when there will be a problem ... this code
 can run for weeks with nothing going wrong, or the deadlock might happen
 only a few minutes after startup.


 Thanks,
 Shawn


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org





-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: Apparent DBCP deadlock related to getConnection

[Phil Steitz]

On 4/11/15 6:29 AM, Shawn Heisey wrote:
 I seem to be having a problem very similar to DBCP-398, using DBCP 2.1
 with MySQL.

 https://issues.apache.org/jira/browse/DBCP-398

 See the last couple of comments.

 I am not familiar with the code, so I can't confirm or refute Phil's
 comment that it looks normal, but I can say this:  The problem came to
 my attention because the program had been deadlocked for several hours,
 with the watchdog thread sending not updating alarms about every 15
 minutes.

 I've been having occasional problems with deadlocks in this program for
 quite a while.  Those problems were occurring even before I switched
 from dbcp to dbcp2.  Until two days ago, I had not tracked down the
 cause of the deadlock.  I was suprised to see that the problem wasn't in
 my own code.

 What other information can I provide to determine whether it's a bug or
 something I've done wrong?

Most likely what is happening is that your application is exhausting
the pool of connections.  The relevant lines in the thread dump you
posted on the ticket are:

at
org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)

What is going on in that thread is that it is blocked waiting for a
connection to become available in the pool.  There is no evidence of
deadlock there - just pool exhaustion.

The most likely culprit is application code that is failing to close
connections that have been borrowed from the pool.  If you don't
close the connections on all execution paths, they will leak as
the pool has no way of knowing that your code is finished with them
unless and until you close them.  What I would do first is carefully
review the code that borrows, uses and returns connections.  Make
sure it always closes them (including exception paths).  If you
can't find the leak, then consider turning on abandoned connection
cleanup (see the DBCP web page for how to configure that).

Phil

 Thanks,
 Shawn


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org





-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

[ANNOUNCEMENT] Apache Commons DBCP 2.1 released

[Phil Steitz]

The Apache Commons Team is pleased to announce the release of Apache Commons 
DBCP 2.1.

The Apache Commons DBCP open source software library implements database 
connection pooling. Version 2.1 is a minor release,
including bug fixes and enhancements. Full details of all the changes in 
version 2.1 can be found in the changelog:
http://commons.apache.org/dbcp/changes-report.html

Source and binary distributions are available for download from the Apache 
Commons download site:
http://commons.apache.org/dbcp/download_dbcp.cgi

When downloading, please verify signatures using the KEYS file available at the 
above location.

For complete information on Commons DBCP, including instructions on how to 
submit bug reports, patches, or suggestions for improvement, see the Apache 
Commons DBCP website:
http://commons.apache.org/dbcp

Phil Steitz, on behalf of the Apache Commons community



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [POOL2] Pooling mutable objects

[Phil Steitz]

 On Feb 7, 2015, at 2:23 AM, Michael Osipov 1983-01...@gmx.net wrote:
 
 Am 2015-02-06 um 19:21 schrieb Phil Steitz:
 On 2/6/15 10:16 AM, sebb wrote:
 The existing Pool2 implementations use equals() to decide if two
 objects are the same.
 
 As of pool 2.3.
 
 You have to make sure that your equals() implementation returns true
 if and only if the objects being compared are the same as far as your
 use of them is concerned.
 So for example two Integers are equal if they have the same integral
 value, but two different StringBuffer instances are never equal.
 
 Once you have determined what constitutes equality for your instances,
 make sure that the hash code is coded accordingly, i.e. two objects
 that are equal() must have the same hash code.
 If this requirment is not met, then the HashMap used internally by
 Pool2 will behave unpredictably.
 
 Also, hash codes should not change between the time that an object
 is borrowed from the pool and subsequently returned.  Again, for
 pool 2.0-2.3 (latest as of this post), objects under management by
 the pool are kept in a hashmap keyed on the objects themselves, so
 mutability that changes hashcodes (and / or equality) can cause
 problems.  See the discussion on POOL-284.
 
 If this causes so many discussions and fuzz, it is worth filing another issue 
 which should document how a pooled object should like. Required criteria. 
 That would have spared me the searching.
 

Definitely fixing javadoc and web site doco will be part of resolution of 
POOL-283/4.

Phil


 WDYT?
 
 Michael
 
 
 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org
 

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [POOL] Release 2.3 in JIRA

[Phil Steitz]

On 2/8/15 1:44 PM, Michael Osipov wrote:
 Hi,

 can anyone kindly release 2.3 in JIRA. It is still in the roadmap
 and not in the changelog.

Done.  Thanks for pointing this out.  The changelog we pay attention
to / keep always up to date, is in /src/changes/changes.xml, which
appears on the website at [1].

Phil

[1] http://commons.apache.org/proper/commons-pool/changes-report.html

 Michael

 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [POOL2] Pooling mutable objects

[Phil Steitz]

On 2/6/15 10:16 AM, sebb wrote:
 The existing Pool2 implementations use equals() to decide if two
 objects are the same.

As of pool 2.3.

 You have to make sure that your equals() implementation returns true
 if and only if the objects being compared are the same as far as your
 use of them is concerned.
 So for example two Integers are equal if they have the same integral
 value, but two different StringBuffer instances are never equal.

 Once you have determined what constitutes equality for your instances,
 make sure that the hash code is coded accordingly, i.e. two objects
 that are equal() must have the same hash code.
 If this requirment is not met, then the HashMap used internally by
 Pool2 will behave unpredictably.

Also, hash codes should not change between the time that an object
is borrowed from the pool and subsequently returned.  Again, for
pool 2.0-2.3 (latest as of this post), objects under management by
the pool are kept in a hashmap keyed on the objects themselves, so
mutability that changes hashcodes (and / or equality) can cause
problems.  See the discussion on POOL-284.

These problems do not impact earlier (1.x) versions of Commons Pool.


 If all RawSession instances are to be regarded as unique (which seems
 likely), then define equals and hash code as noted in POOL-283:

 https://issues.apache.org/jira/browse/POOL-283?focusedCommentId=14307637page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14307637


 On 6 February 2015 at 16:34, Mat Jaggard matt...@jaggard.org.uk wrote:
 He didn't create his own hash function, he generated it in eclipse. This is
 better because you get all the testing of other eclipse users without any
 runtime dependencies.

 On 6 February 2015 at 16:16, William Speirs wspe...@apache.org wrote:

 I'm not sure I understand your comment Mat. From what I can tell, Michael
 made his own hash function. That's fine, but *might* have unexpected
 collision issues. Might not too... I don't know. My guess though is that
 he's not an expert (who is an expert on hash functions?) and might
 unknowingly fall into a trap here.

 For example, commons-lang HashBuilder uses 37 as the prime and 17 as the
 initial result... Michael didn't. Maybe he's OK, maybe he's not... but why
 risk it? Why not just use the library and not reinvent the wheel? (Most
 valid reason would be not wanting to pull in a whole library for these two
 functions/classes.)

 Again, take my advice for what you paid, nothing. :-)

 Bill-

 On Fri, Feb 6, 2015 at 11:06 AM, Mat Jaggard matt...@jaggard.org.uk
 wrote:

 William - I'm not sure if you noticed, but they didn't hand-make
 anything.
 On 6 February 2015 at 16:00, William Speirs wspe...@apache.org wrote:

 Why wouldn't you use the HashCodeBuilder? Much simpler:

 return new HashCodeBuilder().append(internalId).toHashCode();

 Done in 1 line, and probably fewer collisions than your hand-made one.
 Same
 with the EqualsBuilder...

 If you don't want the dep on commons-lang, then I can understand... but
 I'd
 just borrow their internal algo.

 My $0.02...

 Bill-

 On Fri, Feb 6, 2015 at 10:54 AM, Michael Osipov 1983-01...@gmx.net
 wrote:

 This is what I did:
 this.internalId = RandomStringUtils.randomAlphanumeric(8);

 Some Eclipse magic:
 @Override
 public int hashCode() {
 final int prime = 31;
 int result = 1;
 result = prime * result + ((internalId == null) ? 0 :
 internalId.hashCode());
 return result;
 }

 @Override
 public boolean equals(Object obj) {
 if (this == obj)
 return true;
 if (obj == null)
 return false;
 if (getClass() != obj.getClass())
 return false;
 RawSession other = (RawSession) obj;
 if (internalId == null) {
 if (other.internalId != null)
 return false;
 } else if (!internalId.equals(other.internalId))
 return false;
 return true;
 }

 Gesendet: Freitag, 06. Februar 2015 um 16:47 Uhr
 Von: James Carman ja...@carmanconsulting.com
 An: Commons Users List user@commons.apache.org
 Betreff: Re: [POOL2] Pooling mutable objects

 Or just let your IDE generate the methods.


 On Fri, Feb 6, 2015 at 9:05 AM, William Speirs wspe...@apache.org
 wrote:
 I'd think adding a UUID then overriding equals and hashCode would
 do
 the
 trick. To aid you in doing this, commons-lang has EqualsBuilder
 [1]
 and
 HashCodeBuilder [2], I highly recommend using them.

 Bill-


 [1]

 https://commons.apache.org/proper/commons-lang/javadocs/api-3.3.2/org/apache/commons/lang3/builder/EqualsBuilder.html
 [2]

 https://commons.apache.org/proper/commons-lang/javadocs/api-3.3.2/org/apache/commons/lang3/builder/HashCodeBuilder.html
 On Fri, Feb 6, 2015 at 9:00 AM, Michael 

Re: [dbcp] SQLException: Error preloading the connection pool after upgrade to commons-dbcp2

[Phil Steitz]

On 1/27/15 5:56 AM, Cuenca, Sergio wrote:
 TITLE: SQLException: Error preloading the connection pool after upgrade to 
 commons-dbcp2
 ENVIRONTMENT:

   *   Windows XP
   *   Junit 4.11
   *   Java 1.7
   *   Spring 4.0
   *   maven-compiler-plugin 3.2
   *   maven.surefire.plugin 2.17
   *   maven-cobertura-plugin 2.6
   *   Jenkins 1.549
 RELATED ISSUE: https://issues.apache.org/jira/browse/DBCP-431?filter=-2
 DESCRIPTION:

 Hi all,
 We have a continuous integration environment (Jenkins) to launch our JUnits 
 tests.
 We had the version 1.4 of commons-dbcp to create the connection pools with 
 Spring. The configuration was the following:

 bean id=dataSourceTemplate class=org.apache.commons.dbcp.BasicDataSource 
 destroy-method=close
   property 
 name=driverClassNamevalue${aplicacion.datasource.driverClassName}/value/property
   property 
 name=urlvalue${aplicacion.datasource.url}/value/property
   property 
 name=usernamevalue${aplicacion.datasource.user}/value/property
   property 
 name=passwordvalue${aplicacion.datasource.password}/value/property
   property name=initialSizevalue1/value/property
   property name=maxActivevalue15/value/property
   property name=maxIdlevalue2/value/property
   property 
 name=accessToUnderlyingConnectionAllowedvalue${aplicacion.datasource.accessToUnderlyingConnectionAllowed}/value/property
 /bean

 We use JUnit 4.11 and maven cobertura plugin 1.6. to launch the tests.
 This configuration was working CORRECTLY.
 After upgrade to commons-dbcp2, version 2.0.1. Some unit tests are reporting 
 errors with the following message:
 java.sql.SQLException: Error preloading the connection pool.
 We launch one unit test each time, one by one. We create and destroy the 
 connection pool every time.
 What can we do to solve this problem??

Are you sure the tests are not running concurrently?   See comment
on the JIRA.  I think there may be a bug lurking here.

Phil

 Thank you very much.


 Sergio Cuenca Guirado
 RD Department
 Logister, S.A. - GRIFOLS
 Polígon Llevant
 C/Palou, 6
 08150 Parets del Vallès
 Tel: + 34 93 571 02 78
 Fax: + 34 93 571 02 94

 P  Do you need to print this message? Let's protect the environment.


 La información contenida en el presente e-mail es confidencial y está 
 reservada para el uso exclusivo de su destinatario. Se prohíbe estrictamente 
 la distribución, copia o utilización de esta información sin el previo 
 permiso de su destinatario. Si usted no fuera el destinatario, por favor 
 notifíquelo inmediatamente al remitente y elimine el presente mensaje de su 
 sistema informático.

 Information contained in this e-mail is confidential and is intended for the 
 use of the addressee only. Any dissemination, distribution, copying or use of 
 this communication without prior permission of the addressee is strictly 
 prohibited. If you are not the intended addressee, please notify the sender 
 immediately by reply and then delete this message from your computer system.



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [math] About Math.round

[Phil Steitz]

On 1/14/15 6:25 AM, ntysdd wrote:
 Hi,
   I found Oracle has changed the specification of Math.round and
 StrictMath.round.
   See http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6430675
   These methods had surprising behavior for some values. (The bug
 report didn't mention that, but these methods would fail also on large
 integers. I believe Java 7 still have this bug.)
   Does commons math plan to change the behavior?

Thanks for sharing this.  I just opened [1] to track the issue.  We
can decide to resolve as WONT_FIX if we want; but I think we should
follow the BZ report and on first read I am inclined to try to fix it.

Phil

 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [math] About Math.round

[Phil Steitz]

On 1/17/15 10:55 AM, Phil Steitz wrote:
 On 1/14/15 6:25 AM, ntysdd wrote:
 Hi,
   I found Oracle has changed the specification of Math.round and
 StrictMath.round.
   See http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6430675
   These methods had surprising behavior for some values. (The bug
 report didn't mention that, but these methods would fail also on large
 integers. I believe Java 7 still have this bug.)
   Does commons math plan to change the behavior?
 Thanks for sharing this.  I just opened [1] to track the issue.  We
 can decide to resolve as WONT_FIX if we want; but I think we should
 follow the BZ report and on first read I am inclined to try to fix it.

 Phil

Sorry, forgot the URL

[1] https://issues.apache.org/jira/browse/MATH-1196
 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [Math] Statistics summary

[Phil Steitz]

On 1/9/15 8:01 AM, Gilles wrote:
 Hi.

 What is the suggested way to obtain the statistics summary
 of an _array_ of values?

 There is SummaryStatistics in o.a.c.m.stat.descriptive
 but one need to explicitly write the loop to call addValue
 for each element.
 [Shouldn't there be a constructor that takes a double[]
 argument?]

+0 for that - see comment below.

 There are the methods StatUtils but I suspect that it's
 not the most efficient way when several values must be
 retrieved (mean, variance, ...). [Moreover, no method
 exists to directly return the standard deviation.]

Use DescriptiveStatistics.  It has a constructor that takes an
initial array.  That class is designed for the use case in which the
full set of values is stored in memory.  SummaryStatistics is for
streams of values not retained in memory.

Phil


 Best regards,
 Gilles


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

[ANNOUNCEMENT] Apache Commons Pool 2.3 released

[Phil Steitz]

The Apache Commons Team is pleased to announce the release of Apache
Commons Pool 2.3.

The Apache Commons Pool open source software library provides an
object-pooling API and a number of object pool implementations.

No client code changes are required to migrate from any of the 2.x
versions to 2.3. Users of version 1.x should consult the migration
guide on the Commons Pool web site.

Source and binary distributions are available for download from the
Apache Commons download site:
http://commons.apache.org/proper/commons-pool/download_pool.cgi

When downloading, please verify signatures using the KEYS file
available at the above location.

Full details of all the changes in 2.3 can be found in the changelog:
http://commons.apache.org/proper/commons-pool/changes-report.html

For complete information on Commons Pool, including instructions on
how to submit bug reports, patches, or suggestions for improvement,
see the Apache Commons Pool website:
http://commons.apache.org/proper/commons-pool/

Phil Steitz, on behalf of the Apache Commons community


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [Math] Cannot download 3.4

[Phil Steitz]

On 12/22/14 7:07 PM, Harshit Dwivedi wrote:
 Hi,

 I am unable to download Apache Commons Math 3.4 from any of the mirrors. Can 
 you suggest when the latest jar will be available?

The 3.4 release is in progress.  We are reviewing release
candidates, making changes to incorporate feedback and voting to
approve the release.  This sometimes takes a little while.  Once we
have a successful vote, you will see a release announcement here and
the release will be available on the mirrors.

Phil

 Best,
 Harshit Dwivedi
 
 Disclaimer:

 The information contained in this communication is intended solely for the 
 use of the individual or entity to whom it is addressed and others authorized 
 to receive it. This e-mail message may contain confidential, proprietary or 
 legally privileged information. It should not be used by anyone who is not 
 the original intended recipient. The recipient acknowledges that Mirae Asset 
 or its subsidiaries and associated companies, (collectively Mirae Asset 
 Group), are unable to exercise control or ensure or guarantee the integrity 
 of/over the contents of the information contained in e-mail transmissions and 
 further acknowledges that any views expressed in this message are those of 
 the individual sender and no binding nature of the message shall be implied 
 or assumed unless the sender does so expressly with due authority of Mirae 
 Asset Group. If you are not the intended recipient you are hereby notified 
 that any disclosure, copying, distribution or taking any action in reliance 
 on the contents of this information is strictly prohibited and may be 
 unlawful. If you have received this communication in error, please notify us 
 immediately by responding to this email and then delete it from your system. 
 Mirae Asset is neither liable for the proper and complete transmission of the 
 information contained in this communication nor for any delay in its receipt. 
 Nothing in the message is capable or intended to create any legally binding 
 obligations on either party. The information contained in this communication 
 does not constitute a representation that any investment strategy is suitable 
 or appropriate for an investor's individual circumstances. Further, this 
 communication should not be regarded by investors as a substitute for 
 independent professional advice or the exercise of their own judgment. The 
 information contained in this document is for information purposes only and 
 does not constitute any recommendation, offer or solicitation to buy, sell or 
 subscribe to any securities or financial instruments in any jurisdiction. 
 Investors are advised to seek independent professional advice before making 
 any investment. This communication may include hyperlinks to other Mirae 
 Asset Group websites for your convenience. The products and services offered 
 on these websites may be limited to persons located or residing in only that 
 particular jurisdiction. In addition, the content on these linked websites 
 may not be intended for persons located or residing in jurisdictions that 
 restrict the distribution of such content. The terms and conditions governing 
 the use of the website of each Mirae Asset Group entity may differ and you 
 should consult and carefully read the applicable terms and conditions before 
 using the website. Mirae Asset Group accepts no liability for any loss or 
 damage of any kind resulting out of the unauthorized use of this 
 communication.

 Computer viruses can be transmitted via email. The recipient should check 
 this email and any attachments for the presence of viruses and defects. While 
 Mirae Asset has put checks in place to minimize the risks, the company 
 accepts no liability for any damage caused by any virus transmitted by this 
 email.




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

[pool] [dbcp] Re: evictor destroying connections below minIdle?

[Phil Steitz]

On 10/17/14 5:38 PM, Anthony Biacco wrote:
 I just started running pool 2.2 alongside dbcp 2.0.1 in tomcat 7 against
 mysql.

 I noticed a situation while running the idle evictor where there's a time
 during eviction where the number of connections drops below the value set
 for minIdle, so in theory if all connections were idle for
 minEvictableIdleTimeMillis, i could have no free connections.
 After the eviction, connections seem to be created back up to minIdle.

 I expected it to only evict down to the minIdle value, not evict all
 eligible idle connections and then recreate up to minIdle.
 The way i'm viewing the number of connections is through mysql's show full
 processlist command.
 Is what i'm describing possible or am I missing something?

Yes, this is possible.  When evicting idle instances due to
minEvictableIdleTimeMillis exceeded, the evictor does not take into
account the number of idle instances remaining in the pool and it
performs the evict, ensure min idle actions in sequence.  If you
want to avoid the destroy / recreate scenario, you can use
softMinEvictableIdleTimeMillis instead.  
 If it is possible, couldn't this cause the resource to become unavailable
 in the time between destruction and recreation?

I am not sure what you mean here.  This could lead to the idle
instance count hitting zero, but client threads will still get
served as long as maxTotal is not exceeded.  Client threads will
create instances to serve requests when the pool is empty, as long
as there is capacity to do so, and evictor destroy actions
immediately create capacity.  Borrowers may have to incur makeObject
(create physical connection in DBCP case) latency, but they will get
served.

Phil



 Here's my config:
 initialSize=15
 minIdle=15
 maxIdle=100
 maxTotal=200
 maxWaitMillis=4000
 defaultQueryTimeout=20
 testOnBorrow=true
 testWhileIdle=true
 blockWhenExhausted=true
 numTestsPerEvictionRun=50
 timeBetweenEvictionRunsMillis=12
 minEvictableIdleTimeMillis=119000
 validationQuery=SELECT 1
 validationQueryTimeout=2
 removeAbandoned=true
 removeAbandonedOnBorrow=true
 removeAbandonedTimeout=60
 logAbandoned=true

 Thanks,

 -Tony




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [math] JDBC output to generate statistical results.

[Phil Steitz]

On 8/26/14 8:37 AM, Kulpanowski, David wrote:
 Messrs. Worden and Eckenfels:

 Thank you both for your kind assistance. 

 Mr. Worden: 
 your solution works perfectly. This is exactly what I am looking for. 

 Mr. Eckenfels:
 Please excuse my lack of java coding skills. I am working on it by taking on 
 projects at my job. I think your solution will work and I want to use it in 
 my code because I am now going to use Apache Commons Math for more 
 sophisticated statistics such as regression and hypothesis testing. For 
 example, is the mean average ambulance response time in Cape Coral the 
 statistically significantly different from the mean average response time in 
 Fort Myers. I anticipate needing your code so I need to ask for additional 
 help:

 In the final line of code Eclipse is putting a red underline under 
 ArrayUtils. 

 ArrayListDouble times = new ArrayList();
 while (rset.next())
 {
 times.add(Double.valueOf(rset.getDouble(M_SecondsAtStatus)));
 }
 double timesArray[] = ArrayUtils.toPrimitive(times.toArray());

 My mouse hovers over it and the message is: ArrayUtils cannot be resolved. 
 Eclipse offers nine quick fixes:
 1.) create class ArrayUtils.
 2.) create constant ArrayUtils
 3.) create local variable ArrayUtils
 4.) change to ArgUtils
 5.) change to Array
 6.) change to Arrays
 7.) create field ArrayUtils
 8.) create parameter ArrayUtils
 9.) fix project set up

 Which one should I use to output my data in a format Apache Commons Math will 
 utilize in its functions?

Are you perhaps trying to use ArrayUtils from commons lang?  In that
case, you need to import that class from lang.  But as Brent
suggested, it would be easier for you in this case not to build the
list of values at all, but just add them to a DescriptiveStatistics
(or SummaryStatistics) instance that will accumulate stats based on
the stream of values that you give it using addValue(double).  The
difference between DescriptiveStatistics and SummaryStatistics is
that the first one stores the full dataset in memory, so can provide
a few more statistics.  SummaryStatistics does not store the full
data array so can be used with very large samples.

Phil


 -Original Message-
 From: Brent Worden [mailto:brent.wor...@gmail.com] 
 Sent: Tuesday, August 26, 2014 11:00 AM
 To: Commons Users List
 Subject: Re: [math] JDBC output to generate statistical results.

 Another alternative is to use a
 org.apache.commons.math3.stat.descriptive.DescriptiveStatistics
 object to collect all the data and then use it to compute the summary 
 statistics you need.  Using it alleviates the need for doing all explicit 
 type casting and conversion:

 DescriptiveStatistics ds = new DescriptiveStatistics();
 while(rset.next()) {
 int observation = rset.getInt(M_SecondsAtStatus);
 ds.addValue(observation);
 }

 System.out.println(min:  + ds.getMin());
 System.out.println(max:  + ds.getMax()); ...

 HTH,

 Brent


 On Tue, Aug 26, 2014 at 9:41 AM, Bernd Eckenfels e...@zusammenkunft.net
 wrote:

 Hello,

 First of all: Your DBMS might have SQL methods to calculate typical 
 aggregates. This is not only easier to program, but also most likely 
 faster and less resource intensive than doing it in an extra application.

 But since this is the commons list: If You want to use the Commons 
 Math functions you have to present the set of values (in your case as an 
 array).
 And since there is no adapter for result sets (I think) building the 
 array would be done inside the loop. The most natural thing is to use 
 an ArrayList to append the values in the loop,  but then you have to 
 convert the resulting Double[] into double[]. The ArrayUtils in Apache 
 Commons Lang could do that (but if you need to process millions of 
 numbers it is not the most efficient way to do it).

 untested:

 ArrayListDouble times = new ArrayList();
 while(rset.next()) {
   times.add(Double.valueOf(rset.getDouble(T));
 }
 double timesArray[] = ArrayUtils.toPrimitive(times.toArray());

 And then you can use this array for the Math statistics.

 Gruss
 bernd


 --
 http://bernd.eckenfels.net

 - Ursprüngliche Nachricht -
 Von: Kulpanowski, David dkulpanow...@leegov.com
 Gesendet: ‎26.‎08.‎2014 15:55
 An: Commons Users List user@commons.apache.org
 Betreff: RE: [math] JDBC output to generate statistical results.

 Thank you Mr. Ritter:

 Two issues:
 1.) I am attempting to obtain univariate statistics from thousands of 
 ambulance responses. For example, ambulance responses (in seconds) 
 534, 678, 943, 194 would be a mean of 587 seconds. Not by row, but 
 rather as summary statistics.
 2.) It appears that Apache Commons Math is needing a Double value. So 
 I change it as shown below.
 Note on 2) Even though I am needing summary statistics I move the 
 lines of code into the loop just to see what would happen.I just want 
 to get it to work because it appears the problem is the type of 
 variable (int, double, array).

 while (rset.next())

Re: [dbcp] How to create replication aware connection pool ?

[Phil Steitz]

On 8/6/14 1:15 AM, Mark Thomas wrote:
 On 06/08/2014 01:34, Parth Patil wrote:
 Hi Friends,
 I am using DBCP2 in my project and I am pleased with the performance of the
 library. Though I am only able to provide a single mysql host in the
 connection string. What do I need to do inorder to make the DBCP connection
 pool replication aware ?
 Write some code :). DBCP 2 is not replication aware. There was an
 enhancement request for this [1] but I resolved it as WONTFIX on the
 bases that:
 quote
 ...generally, failover requires some form of database replication and
 databases that provide that tend to provide JDBC drivers that support
 failover as well.
 /quote

 I am going to use 1 master and 2 slaves in my
 setup. Am I correct in assuming that its possible to create connection pool
 over several mysql hosts ?
 No, you are not correct.

 I tried to use the replication aware mysql jdbc driver
 (com.mysql.jdbc.ReplicationDriver)
 That is the correct way to do this.

 but that doesn't seem to work and I am
 getting an exception. Following is my test code
 snip/

 Following is the exception that I am getting :

 [error] (run-main-0) java.lang.AbstractMethodError:
 com.mysql.jdbc.ReplicationConnection.isValid(I)Z
 snip/

 DBCP2 uses JDBC4 (Java 1.6) and that is a new method introduced in that
 release.

I think you mean JDBC 4.1, Java 1.7 for DBCP2.

Phil

 You can avoid the call to that method by setting a validation query for
 your connection pool. For MySQL the following should work:
 /* ping */ SELECT 1


 Mark

 [1] https://issues.apache.org/jira/browse/DBCP-393

 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: Commons Config: Variable Interpolation

[Phil Steitz]

On 7/12/14, 9:52 AM, Oliver Heger wrote:
 Hi Ronald,

 I have not used ExprLookup myself, so I am not sure whether I am a big
 help here.

 Your declaration of the lookup seems to be correct, otherwise an
 exception would be thrown by DefaultConfigurationBuilder.

 A difference between your code and the example in the user's guide is
 that in the latter the expression lookup is directly used in the file
 read by DefaultConfigurationBuilder and not in one of the included
 configuration files. For version 2.0 (which is work in progress) this
 should be supported, but I am not sure whether this also works in 1.x.

 I would recommend to start with a simple example first. Test whether you
 can access the lookup directly in this xml file, e.g. by setting the
 file name of a configuration source as demonstrated in the example. By
 this you can find out the results of variable substitutions and compare
 them with your expectations. Only if this works as expected, try again
 to use the lookup in an included configuration source.

I also have no experience using this feature, but IIUC you are
trying to use expr as a prefix.  According to the doc, that is not
among the supported prefixes.

Phil

 HTH
 Oliver

 Am 11.07.2014 17:27, schrieb DiFrango, Ronald (CONT):
 Here's the full XML file:

 ?xml version=1.0 encoding=ISO-8859-1 ?

 configuration
  header
  result /
  lookups
  lookup config-prefix=expr
  
 config-class=org.apache.commons.configuration.interpol.ExprLookup
  variables
  variable name=net 
 value=Class:java.net.InetAddress /
  /variables
  /lookup
  /lookups
  /header

  system /

  properties fileName= example.properties listDelimiter=|
  reloadingStrategy refreshDelay=1
  
 config-class=org.apache.commons.configuration.reloading.FileChangedReloadingStrategy
  /
  /properties

 /configuration

 And the referenced properties file:

  mobile.rest.host=${expr:$[net.localHost.hostName]}


 Ron DiFrango



 -Original Message-
 From: Martin Gainty [mailto:mgai...@hotmail.com] 
 Sent: Friday, July 11, 2014 11:25 AM
 To: Commons Users List
 Subject: RE: Commons Config: Variable Interpolation



 From: ronald.difra...@capitalone.com
 To: user@commons.apache.org
 Date: Fri, 11 Jul 2014 11:02:45 -0400
 Subject: RE: Commons Config: Variable Interpolation

 Just to complete it, the code looks like:

 DefaultConfigurationBuilder factory = new 
 DefaultConfigurationBuilder(example.xml);
 Configuration config = factory.getConfiguration();
 MGno clue which properties are defined in example.xml so lets add a new 
 MGproperty called mobile.rest.host here 
 MGconfig.addProperty(mobile.rest.host, maps.google.com);

 System.out.println(Host Name =  + 
 config.getString(mobile.rest.host));
 MG?
 Ron DiFrango


 -Original Message-
 From: DiFrango, Ronald (CONT) [mailto:ronald.difra...@capitalone.com]
 Sent: Friday, July 11, 2014 10:49 AM
 To: user@commons.apache.org
 Subject: Commons Config: Variable Interpolation

 I'm attempting to perform Variable Interpolation as outlined here:

 http://commons.apache.org/proper/commons-configuration/userguide/howto
 _basicfeatures.html#Variable_Interpolation

 I'm using an XML file as the starter so I added the following blurb to the 
 beginning of the file:

 header
   result /
   lookups
  lookup config-prefix=expr

 config-class=org.apache.commons.configuration.interpol.ExprLookup
variables
   variable name=net 
 value=Class:java.net.InetAddress /
/variables
  /lookup
   /lookups
/header
 MG
 override
   xml mobileRestHost=String.lowercase(net.localHost.hostName)/xml
 /override
 MG?

 And then in my properties file that is loaded I did:

 mobile.rest.host=${expr:$[net.localHost.hostName]}

 But it did not work any thoughts on what I'm doing wrong?

 Thanks,

 Ron


 

 The information contained in this e-mail is confidential and/or proprietary 
 to Capital One and/or its affiliates. The information transmitted herewith 
 is intended only for use by the individual or entity to which it is 
 addressed.  If the reader of this message is not the intended recipient, 
 you are hereby notified that any review, retransmission, dissemination, 
 distribution, copying or other use of, or taking of any action in reliance 
 upon this information is strictly prohibited. If you have received this 
 communication in error, please contact the sender and delete the material 
 from your computer.
 

 The information contained 

Re: Could we have a roots() method in PolynomilFunction class?

[Phil Steitz]

 On Jul 3, 2014, at 2:30 PM, Gilles gil...@harfang.homelinux.org wrote:
 
 On Thu, 3 Jul 2014 18:14:41 +0200, Axel wrote:
 Ok,
 but what about my main question:
 Could we have a roots() method in PolynomialFunction class?
 
 Which in the first step delegates to LaguerreSolver#solveAllComplex()?
 
 I guess that people want to be careful before changing the API of
 PolynomialFunction. [E.g. it would create a dependency towards
 the o.a.c.m.complex package. It might be better to add the
 functionality in that package (e.g. in ComplexUtils).]
 
 Could you explain what you need with more details?
 In particular, what do you expect to get in addition to what the
 following code can already provide?
 
 ---CUT---
 import org.apache.commons.math3.analysis.polynomials.PolynomialFunction;
 import org.apache.commons.math3.analysis.solvers.LaguerreSolver;
 import org.apache.commons.math3.complex.Complex;
 
 public class PolynomialRoots {
private static final LaguerreSolver solver = new LaguerreSolver();
 
public static Complex[] solve(PolynomialFunction p) {
return solver.solveAllComplex(p.getCoefficients(), 0d);
}
 }
 ---CUT---

I agree with Thomas that it would be good to expose a Complex[] roots() method 
directly in the PolynialFunction class.  We can then choose whatever numerical 
technique we like to back it, starting with Laguerre and refining / 
specializing to data as we get better ideas.

Phil
 
 
 Best regards,
 Gilles
 
 
 
 On Sat, Jun 28, 2014 at 8:26 PM, Ted Dunning ted.dunn...@gmail.com wrote:
 That is one article, but it doesn't actually compare the numerical
 stability or efficiency of this method.  It just invokes the stability of
 numerical linear algebra.  Whether this is a good way to compute roots is
 an open question.
 
 The other major question here is operation count.  Computing eigenvalues of
 an explicit matrix is likely to be very intensive computationally.  The
 wikipedia page on root-finding mentions in passing when is says that
 matrix-free methods are typically used with the eigenvalue approaches.
 
 This suggests that the preferable means to implement this idea is not to
 build the matrix in question, but to use an abstract linear operator which
 implements multiplication making use of the special form of the companion
 matrix.  I am not sure if this approach is viable in the Commons matrix
 framework.  I suspect not, but really don't have much of a clue about the
 real state of things.  If the eigenvalue objects accept something like a
 LinearOperator object, then it is likely to work.
 
 This article[1] seems to suggest that there may be some numerical issues
 with large coefficients.  That isn't surprising since many algorithms have
 similar problems.
 
 [1]
 http://noether.math.uoa.gr/conferences/sla2014/sites/default/files/Dopico.pdf
 
 
 On Sat, Jun 28, 2014 at 6:33 AM, Axel axel...@gmail.com wrote:
 
 On Fri, Jun 27, 2014 at 10:56 PM, Thomas Neidhart
 thomas.neidh...@gmail.com wrote:
 ...
  I did take a look at the stackoverflow question, and there is already a
  way to do this in Commons Math using the LaguerreSolver via the
  solveComplex and solveAllComplex methods.
 
  But it might be good to support a second way using EigenDecomposition as
  a stand-alone solver.
 
  I like the idea to add a roots() method to PolynomialFunction, but which
  method to compute the roots is more robust?
 
 The attached link in the stackoverflow question to this paper:
 http://techdigest.jhuapl.edu/TD/td2804/Williams.pdf
 
 has this conclusion:
 We have discussed some eigenvector methods for finding the roots of multi-
 variate polynomials. Unlike iterative, numerical methods typically
 applied to this
 problem, the methods outlined in this article possess the numerical
 stability of
 numerical linear algebra, do not require a good initial guess of the
 solution, and give all
 solutions simultaneously. Furthermore, if the initial guess is poor
 enough, the methods
 outlined herein may converge more quickly than iterative methods.
 
 So I think the EigenDecomposition method is more appropriate if you
 don't have an initial guess to start from getting the roots!?
 
 --
 Axel Kramer
 
 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org
 
 
 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org
 

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: Multiplexed connections in commons pool

[Phil Steitz]

On 6/19/14, 2:36 PM, James Leskovar wrote:
 Hi Phil,

 Thanks for the reply. I'll try my hand first at the PairConn,Int approach, 
 and if it looks workable I can open up a Jira issue.

Great!  Feel free to ask questions if things don't fall into place
easily. 

Phil

 Cheers,
 James


 -Original Message-
 From: Phil Steitz [mailto:phil.ste...@gmail.com] 
 Sent: Thursday, 19 June 2014 12:24 PM
 To: Commons Users List
 Subject: Re: Multiplexed connections in commons pool

 On 6/18/14, 4:07 PM, James Leskovar wrote:
 Hi there,

  

 I have an application that needs to perform connection pooling, with 
 the proviso that it's okay - and actually preferable - for more than 
 one client to checkout and use the same connection object from the 
 pool. Ideally, I would also like to limit the number of concurrent 
 clients that are using a single connection object. I'm wondering what 
 the best way to do this is. As a quick and dirty option, I suppose I 
 could basically have my PooledObjectFactory return the same objects 
 from makeObject(), and manually keep track of objects from inside my 
 implementation.
 Thoughts?

 This is an interesting use case that unfortunately is that easy to support in 
 [pool].  If you are interested in helping design and implement direct 
 support, please feel free to hop over to the dev list and open a JIRA.

 Your idea of just having the factory return identical instances won't work 
 because when clients return them, they will get exceptions because repeated 
 returns of the same object violates the
 pool contract.   You can make that approach work though by having
 your factory source Foo, int pairs where Foo is what you want to pool.  
 Just make sure the FooInt instances are distinguishable by equals.  
 Unfortunately, your factory will have to maintain counters, which it can do 
 via the lifecycle methods and probably a hashmap keyed on the actual Foo 
 instances with rep counts as values.

 Phil
  

 Cheers,

 *James Leskovar*
 Software Engineer, Location Platforms

 TeleCommunication Systems, Inc.

 [ *Address* : TCS, iC Enterprise 1, Innovation Campus, Squires Way, 
 Nth Wollongong, NSW, 2500, Australia ] [ *Tel* : +61 2 4221 2940 ] [ 
 *Fax* : +61 2 4221 2901 ] [ *Email* : james.lesko...@telecomsys.com 
 mailto:james.lesko...@telecomsys.com ]

 TCS http://www.telecomsys.com/default.aspxFacebook
 https://www.facebook.com/pages/TCS/340389379402958Twitter
 https://twitter.com/TeleComSysLinkedIn
 http://www.linkedin.com/company/telecommunication-systems

  

 CONFIDENTIALITY NOTICE: The information contained in this message may 
 be privileged and/or confidential. If you are not the intended 
 recipient, or responsible for delivering this message to the intended 
 recipient, any review, forwarding, dissemination, distribution or 
 copying of this communication or any attachment(s) is strictly 
 prohibited. If you have received this message in error, please notify 
 the sender immediately, and delete it and all attachments from your 
 computer and network.


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

[ANNOUNCEMENT] Apache Commons DBCP 2.0.1 released

[Phil Steitz]

The Apache Commons Team is pleased to announce the release of Apache
Commons DBCP 2.0.1.

The Apache Commons DBCP open source software library implements
database connection pooling. Version 2.0.1 is a patch release,
including bug fixes only.

Source and binary distributions are available for download from the
Apache Commons download site:
http://commons.apache.org/proper/commons-dbcp/download_dbcp.cgi

When downloading, please verify signatures using the KEYS file
available at the above location.

Version 2.0.1 is source and binary compatible with version 2.0. 
Full details of all the changes in 2.0.1 can be found in the changelog:
http://commons.apache.org/proper/commons-dbcp/changes-report.html

For complete information on Commons DBCP, including instructions on
how to submit bug reports, patches, or suggestions for improvement,
see the Apache Commons DBCP website:
http://commons.apache.org/proper/commons-dbcp/

Phil Steitz, on behalf of the Apache Commons community

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] - How to programmatically check the pool status?

[Phil Steitz]

On 4/16/14, 2:33 AM, Raffaele Gambelli wrote:
 Hi all,

 I'm using Tomcat 6.0.26 and its default dbcp connection pooling
 system (tomcat-dbcp.jar inside the default installation dir)

 My question is:
 How could I check programmatically the connection pool status ?
 For example inside a catch of some exception related to some
 database issues I would like to log some info like the number of
 currenlty open connections, idle etc...

Have a look at the javadoc for BasicDatasource for the version that
tomcat 6 is using (some 1.x version).  There are getters for the
number of active and idle connections.  You may need to cast the
reference that tomcat gives you to the datasource to use these
methods.  You may want to ask about this on the tomcat user list.

Phil

 Thanks in advance and best regards

 Raffaele Gambelli

 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

[pool][dbcp][scxml] Apachecon, North America April 7-9

[Phil Steitz]

I will be presenting on the newly-released 2.x versions of Commons
Pool and DBCP next month at Apachecon, North America in Denver [1].

Ate Douma is also giving a talk on [scxml].

Come join us!

Register soon, as prices go up on March 14th.

Phil

[1] http://na.apachecon.com/.


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [math] Stat.Regression Class structure/output

[Phil Steitz]

On 2/21/14, 12:39 PM, Zachary Seid wrote:
 Hello all,
 I would like to access information wrapped in the RegressionResults class,
 but for OLSMultipleRegression and/or GLSMultipleRegression.

Great suggestion.
  However,
 OLS/GLS do not currently support the UpdatingMultipleLinearRegression
 interface, and I was wondering if there is a simple workaround to provide
 equivalent content. If not, I was wondering if the current structure was a
 design decision and if there would be opposition in development to
 refactoring OLS/GLS to implement the UpdatingMultipleLinearRegression
 interface.

Do you need the updating (adding data to a model and then
re-estimating parameters) or is just encapsulation of results
sufficient?  The latter would be simple to add to the OLS/GLS
classes.  Patches welcome!

Phil
 Thank you,
 Zachary



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [math] 3.3 release date

[Phil Steitz]

On 1/25/14, 9:37 AM, supa wrote:
 Is version 3.3 release date known ?

No, we don't set fixed release dates.  That said, I think we are
getting close, which means some time in the next month or so.

Phil


 Thanks,
  Siarhei

 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [math] 3.3 release date

[Phil Steitz]

On 1/25/14, 9:37 AM, supa wrote:
 Is version 3.3 release date known ?

No, we don't set fixed release dates.  That said, I think we are
getting close, which means a release is likely some time in the next
month or so.

Phil


 Thanks,
  Siarhei

 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

[ANNOUNCEMENT] Apache Commons Pool 2.1 released

[Phil Steitz]

The Apache Commons Team is pleased to announce the release of Apache Commons 
Pool 2.1.

The Apache Commons Pool open source software library provides an object-pooling 
API and a number of object pool implementations.

No client code changes are required to migrate from version 2.0 to 2.1.  Users 
of version 1.x should consult the migration guide on the Commons Pool web site.

Source and binary distributions are available for download from the Apache 
Commons download site:
  http://commons.apache.org/proper/commons-pool/download_pool.cgi

When downloading, please verify signatures using the KEYS file available at the 
above location when downloading the release.

Full details of all the changes in 2.1 can be found in the changelog:
  http://commons.apache.org/proper/commons-pool/changes-report.html

For complete information on Commons Pool, including instructions on how to 
submit bug reports, patches, or suggestions for improvement, see the Apache 
Commons Pool website:

http://commons.apache.org/proper/commons-pool/

Phil Steitz, on behalf of the Apache Commons community


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

[all] ApacheCon North America 2014

[Phil Steitz]

The Call For Proposals for ApacheCon NA, 2014 will go out in the
next couple of weeks.  PMCs have been asked for volunteers for the
Content Committee to review proposals related to their projects. 
Before we all rush to raise our hands, we thought it best to get a
sense for what talks might be proposed the community.  Is anyone
planning to submit a proposal?  The theme this year is really
wide-open, i.e., there is no pre-established theme for the whole
conference of pre-determined track structure.

Phil

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [pool] Factory example

[Phil Steitz]

On 11/29/13, 7:28 AM, Manuel Irribarra wrote:
 Hi,

 Following the example page, I noted that Factory example for
 PooleableObjectFactory is wrong. When you extends
 BasePooledObjectFactory?, there only 2 methods required:

 + public ? create()
 + public PooledObject? wrap (? object)

 I guess that create method must created a new object instance,
 but... ¿how I must use the wrap method?

 Any one have an example?

Yes, create() creates a new object instances.

wrap creates a PooledObject from a newly-created instance.  There is
a DefaultPooledObject implementation provided.  To use that, just use

@Override
 public PooledObjectFoo wrap(Foo foo) {
return new DefaultPooledObjectFoo(foo);
 }
 
Thanks for pointing out the problem with the example page.  It needs to be 
updated to reflect the new API.  It is still using the old 
PoolableObjectFactory, which no longer exists.  I will replace this with a 
working 2.0 example shortly.

Phil



 -- 
 *Manuel A. Irribarra Frex*
 *Ingeniero de Desarrollo*
 Altiuz Soluciones Tecnológicas de Negocios Ltda.
 Av. Nueva Tajamar 555 Of. 802, Las Condes - CP 7550099
 +56 2 2335 2461
 mirriba...@altiuz.cl mailto:mirriba...@altiuz.cl
 http://www.altiuz.cl
 http://www.altiuzreports.com  https://www.facebook.com/altiuz
 http://twitter.com/altiuz http://www.linkedin.com/company/altiuz

   



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [math] Determinig steep drop (high rate of change) in a set of numbers

[Phil Steitz]

On 11/28/13, 3:39 PM, Mugoma Joseph O. wrote:
 Hello,

 I have a set of numbers arranged in desc order. I would like to determine
 if there's any steep drop and where it occurs i.e. where rate of change is
 high.

 e.g. if I define steep drop (high Rate of Change) as any change more than
 30% then, for the following numbers, the steep drop will be between 4.5 
 2:

 11.9, 10.5, 10, 9.9, 9.5, 8.5, 7.9, 6.5, 6, 5, 4.5, 2, 1.9, 1.5, 1.1, 1

 Is there a function is commons maths that I can use to determine steep drop?

No, there is nothing in [math] any simpler than just walking the
array, computing the changes and comparing against the threshold.

Phil

 Thanks in advance.

 Mugoma.


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [Math] Use of Precision class for inequalities

[Phil Steitz]

On 10/30/13 4:04 AM, Luc Maisonobe wrote:
 Le 30/10/2013 11:45, BIGNON Emmanuel a écrit :
 Hello,
 Hi Emmanuel,

 Commons Math has a class Precision that defines equalities and 
 inequalities using epsilon values.
 I noticed that Commons Math uses the Precison.equals method when necessary 
 (for equalities), but never the Precision.compareTo method (for 
 inequalities).
 Is there a specific reason for that asymmetrical behaviour?
 No, there are no specific reasons. The use of the Precision class
 spreads slowly as we work on various code aspects, but we did not try to
 enforce the change everywhere in one big move.

Right.  Are there specific examples that you have in mind, Emanuel? 
If you see examples where the code as it stands is numerically
incorrect or unstable due to strict inequality tests, please open
JIRA tickets [1].

Thanks for the feedback!

Phil

[1] http://commons.apache.org/proper/commons-math/issue-tracking.html

 best regards,
 Luc

 Emmanuel


 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org

 .



-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [dbcp] does closing a connection return an object to the pool?

[Phil Steitz]

On 9/30/13 10:54 AM, Jonathan Buhacoff wrote:
 Forgot the [dbcp] in the subject on my first post. Reposting:

 I'm using a library that accepts a DataSource as an input and automatically 
 calls getConnection() as needed and then close() on each connection when it's 
 done.

 I want to use a connection pool so I was casually looking at 
 PoolingDataSource in DBCP 1.4 to see if would work. 

 In the source code I see that PoolingDataSource.getConnection() calls 
 borrowObject() on the pool to get a new database Connection, then it wraps it 
 with PoolGuardConnectionWrapper.  But then... the close() method of 
 PoolGuardConnectionWrapper does NOT call returnObject() on the pool!  It 
 simply forwards the close() call to the delegate/wrapped Connection object.  

The delegate is a PoolableConnection.  Its close method returns it
to the pool.

 I wanted to write a quick test to see if this is really an issue or if it's 
 working and I just missed something while browsing the source code.  Looking 
 at the javadocs I noticed that GenericObjectPool needs a 
 PoolableObjectFactory to function, yet PoolableConnectionFactory requires an 
 ObjectPool in its constructor.  
PoolableConnectionFactorys need to create PoolableConnections. 
PoolableConnections need to be attached to pools so their close
methods can return them to the pool that owns them.  Therefore, PCFs
need to hold references to the object pools that manage the
connections that they create.  As you note, object pools also
require factories, so the dependency is two-way.  The PCF
constructor sets itself to be the factory for its pool.
 It seems to me that in  order to implement the PoolableObjectFactory one 
 shouldn't need a reference to the ObjectPool itself.  The pool interfaces 
 seem well defined but the DBCP implementations seem very messy. 

 Can anyone using this library confirm that it's working as expected, not 
 leaking connections?   Just wondering while I set up a test to see for myself.

Please do let us know if you find connection leaks not due do client
abandonment.

Phil

 --
 Jonathan Buhacoff
 jonat...@buhacoff.net
 --






-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: [DBCP]a question about ManualPoolingDriverExample

[Phil Steitz]

Good question, Li Li.  The answer is no, this is done by the
PoolableConnectionFactory constructor.  You can see that in the
source here:
http://svn.apache.org/repos/asf/commons/proper/dbcp/branches/DBCP_1_3_x_BRANCH/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java.

Phil

On 9/28/13 10:09 PM, Li Li wrote:
 hi all
 I am using DBCP as a connection pool using codes like
 http://svn.apache.org/repos/asf/commons/proper/dbcp/branches/DBCP_1_3_x_BRANCH/doc/ManualPoolingDriverExample.java

there is a line:  PoolableConnectionFactory
 poolableConnectionFactory = new
 PoolableConnectionFactory(connectionFactory,connectionPool,null,null,false,true);

it creates a PoolableConnectionFactory but never use it.

do I need add this? connectionPool.setFactory(poolableConnectionFactory);

 -
 To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
 For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: commons pool benchmark issue

[Phil Steitz]

On 9/28/13 11:50 PM, Daniel Wu wrote:
 Hi guys,

 I wrote a simple test for commons-pool. The result shows that the
 performance drops dramatically when concurrent threads increase.
 e.g, With 50 threads, the throughput is 603326 transactions per
 second, when threads increase to 600, the throughput drops to
 32797. The diagram below is the response time in milliseconds,
 you can see as the threads increase, the response time grows quickly.



 I guess my benchmark test code must have something wrong, could
 you guys help me review the test code below and figure out what's
 wrong with my test code?


 public class BenchmarkCommons {

 public BenchmarkCommons(int workerCount, int loop) throws
 Exception {
 double[] statsAvgRespTime = new double[workerCount];
 CountDownLatch latch = new CountDownLatch(workerCount);

 GenericObjectPool pool = new GenericObjectPool(new
 PoolableObjectFactory() {
 @Override
 public Object makeObject() throws Exception {
 return new StringBuilder();
 }
 @Override
 public void destroyObject(Object o) throws Exception {
 }
 @Override
 public boolean validateObject(Object o) {
 return true;
 }
 @Override
 public void activateObject(Object o) throws Exception {
 }
 @Override
 public void passivateObject(Object o) throws Exception {
 }
 });
 pool.setMinIdle(25);
 pool.setMaxIdle(50);
 pool.setMaxActive(50);

 Worker[] workers = new Worker[workerCount];
 for (int i = 0; i  workerCount; i++) {
 workers[i] = new Worker(i, pool, latch, loop,
 statsAvgRespTime);
 }
 long t1 = System.currentTimeMillis();
 for (int i = 0; i  workerCount; i++) {
 workers[i].start();
 }
 latch.await();
 long t2 = System.currentTimeMillis();
 double stats = 0;
 for (int i = 0; i  workerCount; i++) {
 stats += statsAvgRespTime[i];
 }
 System.out.println(Average Response Time: + new
 DecimalFormat(0).format(stats / workerCount));
 System.out.println(Average Througput Per Second: + new
 DecimalFormat(0).format(( (double) loop * workerCount * 1000 )
 / (t2 - t1) ));
 }

 private static class Worker extends Thread {

 private final int id;
 private final GenericObjectPool pool;
 private final CountDownLatch latch;
 private final int loop;
 private final double[] statsAvgRespTime;

 public Worker(int id, GenericObjectPool pool,
 CountDownLatch latch, int loop, double[] statsAvgRespTime) {
 this.id = id;
 this.pool = pool;
 this.latch = latch;
 this.loop = loop;
 this.statsAvgRespTime = statsAvgRespTime;
 }

 @Override public void run() {
 long t1 = System.currentTimeMillis();
 for (int i = 0; i  loop; i++) {
 StringBuilder obj = null;
 try {
 obj = (StringBuilder) pool.borrowObject();
 obj.append(x);
 } catch (Exception e) {
 e.printStackTrace();
 } finally {
 if (obj != null) {
 try {
 pool.returnObject(obj);
 } catch (Exception e) {
 e.printStackTrace();
 }
 }
 }
 }
 long t2 = System.currentTimeMillis();
 statsAvgRespTime[id] =  ((double) (t2 - t1)) / loop;
 latch.countDown();
 }
 }

 public static void main(String[] args) throws Exception {
 System.out.println(---warm up);
 new BenchmarkCommons(50,  100);
 System.out.println(Apache commons pool
 test---);
 new BenchmarkCommons(50,  50);
 new BenchmarkCommons(100, 50);
 new BenchmarkCommons(150, 30);
 new BenchmarkCommons(200, 30);
 new BenchmarkCommons(250, 10);
 new BenchmarkCommons(300, 10);
 new BenchmarkCommons(350, 5);
 new BenchmarkCommons(400, 5);
 new BenchmarkCommons(450, 2);
 new BenchmarkCommons(500, 2);
 new BenchmarkCommons(550, 1);
 new BenchmarkCommons(600, 1);
 }

 }


First some questions, then some observations.

0) What version of Commons Pool are you running?
1) How many physical CPU cores does the machine you are using to run
the test have?

Now some observations:

2) To measure the performance of the pool itself, you should time
the borrows and returns, not the full loop including the client action
3) With a 

Re: [dbcp] GenericObjectPool.borrowObject: Timeout waiting for idle object

[Phil Steitz]

On 9/29/13 7:13 AM, Bosio Andrea wrote:

 Hello all,

 I’m getting the following exception, using commons-dbcp-1.4.

 Caused by: java.util.NoSuchElementException: Timeout waiting for
 idle object

 at
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
 ~[commons-pool-1.5.5.jar:1.5.5]

 at
 org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
 ~[commons-dbcp-1.4.jar:1.4]

 ... 62 common frames omitted

 Connections are managed by spring. These are the dbcp config
 parameters:

 poolPreparedStatements=true

 defaultAutoCommit=false

 driverClassName=oracle.jdbc.driver.OracleDriver

 validationQuery=select 0 from dual

 initialSize=1

 maxActive=15

 maxIdle=15

 maxWait=6

 removeAbandoned=false

 removeAbandonedTimeout=300

 On the database (oracle) I see no active connections when the
 problem occurs, and sometimes a few inactive connections (5
 connections).

 I got an heap dump, and found that the ConnectionPool object
 (instance of org.apache.commons.pool.impl.GenericObjectPool)
 numActive variable value is 15, so I understand that the
 connection pool thinks that there are 15 active connections.

 I can’t understand why this happen, because on the database there
 aren’t so many connections.

 On the heap dump, another thing is strange: the ConnectionPool
 object variable “pool”, which as far I understand is the list of
 the connections managed by the pool, contains no element!


The most likely explanation for this is that connections checked out
from the pool are being abandoned by client threads - i.e., never
returned via the close() method. The first thing to check is that on
all code execution paths (including exceptions), the connections
borrowed from the pool are being closed. You can turn on abandoned
connection removal and logging to verify that this is not happening.
If turning on abandoned connection removal makes the problem go
away, you should still find and fix the code paths that abandon
connections.

Unless client threads are being interrupted, this will not likely
fix anything, but you can try upgrading to version 1.5.7 of Commons
Pool.

Phil

 Bye,
 Andrea



 Andrea Bosio
 Manager

 Technology Reply
 Via Cardinal Massaia, 71
 10147 - Torino - ITALY
 phone: +39 011 29100
 mobile: +39 348 9504513
 a.bo...@reply.it
 www.reply.it

 Technology Reply


 

 --
 The information transmitted is intended for the person or entity
 to which it is addressed and may contain confidential and/or
 privileged material. Any review, retransmission, dissemination or
 other use of, or taking of any action in reliance upon, this
 information by persons or entities other than the intended
 recipient is prohibited. If you received this in error, please
 contact the sender and delete the material from any computer.


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org