RE: commons dbcp or pool problems

[Waldhoff, Rodney]

Martin Cooper wrote:

> What do you have in mind with regard to Digester-based 
> configuration? I'd be willing to take a crack at doing 
> this if you could give me some pointers on what you'd like 
> to see. Bear in mind that I'm not exactly a JOCL expert. ;-)

Recall that an "instance" of DBCP glues together three different tiers:

* A client-interface tier (like PoolingDataSource or PoolingDriver)

* A pool implementation (like GenericObjectPool or anything else that
implements org.apache.commons.pool.ObjectPool)

* A (Poolable)Connection factory (like DataSourceConnectionFactory or
DriverConnectionFactory or DriverManagerConnectionFactory)

One simple (but run-time inflexible) way of doing this is just to glue the
pieces together with a little bit of Java code, as Craig has done with
BasicDataSource (see the createDataSource method), or as shown in the
Manual*Example.java files in the dbcp/doc directory.  

(In retrospect, perhaps we should stress this way of using DBCP a bit
more--providing basic, "pre-assembled" connection pools that are suitable
for most people's needs--which avoids the configuration learning curve.)

In the general case, we'd like (or at least I'd like) to allow run-time or
at least init-time selection and configuration of each tier, so that I can
put together an arbitrary DBCP "instance" without changing the source code.
Hence some configuration mechanism is necessary.  In Java code, it's really
not that hard, a few lines will suffice:

// create a pool
ObjectPool connPool = new GenericObjectPool(...);

// create a connection factory
ConnectionFactory factory = new DriverManagerConnectionFactory(...);

// wrap the factory in a PoolableConnectionFactory
// which will allow the connections created by factory to be pooled
PoolableConnectionFactory poolableConnectionFactory = new
PoolableConnectionFactory(...);

// create a PoolingDataSource from that pool
PoolingDataSource dataSource = new PoolingDataSource(...);

(This example is pulled from setupDataSource in
http://cvs.apache.org/viewcvs/~checkout~/jakarta-commons/dbcp/doc/ManualPool
ingDataSourceExample.java?content-type=text/plain if you'd like to see more
detail.)

The trouble is, if we want generality, we can't know the implementation of
ObjectPool or ConnectionFactory before-hand, and most implementations of
those will have quite a number of configuration options we'll want to
expose.  (And the complexity increases if I want to allow
PreparedStatementPooling, since then I'll need/want to configure the pools
of statements as well.)

JOCL was an effort to solve this problem, although it's clearly more
oriented toward the easiest implementation as opposed to the most-user
friendly configuration.  If it's easy to do the configuration by
instantiating Java objects, then give me a way to instantiate Java objects
via my configuration file.

It works like this: Every element in a JOCL document represents the
instantiation of a Java object (or primitive).  Hence there's an 
tag, one for each primitive (,, etc.) and for convenience, a
 tag.  Object tags have an attribute indicating the class to be
instantiated, and may contain other tags which indicate the signature of the
constructor to be invoked as well as the values to pass into it.  For
example:



is equivalent to:

new java.util.Date(1015542020159L);

and 


 
   
 


is equivalent to:

new mypackage.Foo(new java.util.Date(1015542020159L));

So, whatever you can create via a chain of constructors and primitives, you
can create with JOCL.

I find this works rather well in practice as long as the file is maintained
by someone slightly Java-literate. (See
http://cvs.apache.org/viewcvs/~checkout~/jakarta-commons/dbcp/doc/poolingDri
verExample.jocl.sample?content-type=text/plain for a DBCP example.)  The
main problem is that the JOCL file doesn't provide much context on it's
own--you either need to add XML comments or look up the constructor
signatures and their meanings in the JavaDocs.

In my original comment:

> The JOCL configuration should probably either be 
> converted to Digester or something like it, or JOCL 
> moved to a stand-alone component.

The point I was trying to make is that JOCL is an odd component to have
tucked away inside of DBCP, so let's either promote it or remove it.
Digester and JOCL have a lot of overlap on the surface--both provide a way
of mapping XML to Java methods and both are largely used for supporting
XML-based configuration files.  The primary difference I see between the two
is that Digester seems oriented toward writing small amounts of specialized
Java code for parsing known-at-compile-time DTDs/schemas, while JOCL tries
to provide single syntax for java-object-construction, supporting (however
crudely) the configuration of whatever new types should come along.  (So,
for example, if I create a great new ObjectPool implementation, I can
convert all my DBCP pools to using it by simply adding the new impl to my
classpath, and changing that secti

Re: commons dbcp or pool problems

[Martin Cooper]

- Original Message -
From: "Waldhoff, Rodney" <[EMAIL PROTECTED]>
To: "'Jakarta Commons Developers List '" <[EMAIL PROTECTED]>
Sent: Wednesday, March 06, 2002 4:27 AM
Subject: RE: commons dbcp or pool problems


> > What is the status of commons dbcp?
> > Is this considered done
> > and production quality or it still in beta (or less?)
>
> IMO, commons-dbcp is production quality, indeed, I use it for a number
> of
> production applications.  We could do a 1.0 release pretty much anytime,
> although the documentation is probably insufficient, and the JOCL
> configuration should probably either be converted to digester or
> something
> like it, or JOCL moved to a stand-alone component.

What do you have in mind with regard to Digester-based configuration? I'd be
willing to take a crack at doing this if you could give me some pointers on
what you'd like to see. Bear in mind that I'm not exactly a JOCL expert. ;-)

I have a vested interest in seeing DBCP released soon, because we're going
to be releasing Struts 1.1 soon, and that relies on it...

--
Martin Cooper


>
> > Apart from what look like minor
> > edits from Craig, it's been 10 months
> > (according to CVS) since any major
> > work has been done.
>
> And one could see that as a good thing.
>
> There was some talk on this list a few weeks back about some minor
> enhancments to DBCP/Pool (like having the pool methods throw Exception,
> allowing DBCP to propogate the original SQLExceptions, etc.) but no
> patches
> materialized out of that discussion.
>


--
To unsubscribe, e-mail:   <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>

Re: commons dbcp or pool problems

[Martin Cooper]

- Original Message -
From: "Geir Magnusson Jr." <[EMAIL PROTECTED]>
To: "Jakarta Commons Developers List" <[EMAIL PROTECTED]>
Sent: Wednesday, March 06, 2002 12:16 PM
Subject: Re: commons dbcp or pool problems


> On 3/6/02 2:53 PM, "John McNally" <[EMAIL PROTECTED]> wrote:
>
> >
> > The code in sandbox/jdbc2pool uses the commons pool and DBCP's
> > PreparedStatement pool while attempting to implement the jdbc2
> > specification.  Geir and anyone else at jakarta who wants to work on a
> > connection pool, why not join me in finishing this?  Geir, why are you
> > saying you want to work on Poolman, what does it have that dbcp does
> > not?
>
> I am saying that I think Poolman is worth saving.  I rely on it in several
> client projects, and I am sure there are many users of Poolman here,
despite
> the existence of the dbcp.
>
> This is not a reflection on DBCP, but rather acting on the knowledge that
> Poolman has a huge user base and a long track record of reliability.

I agree that Poolman is worth saving. From what I can tell from various
mailing lists I am on, it has a substantial and enthusiastic following. A
big part of that is because it's recognised as being reliable, good quality
software. IMHO, that makes it a good candidate for Jakarta. ;-)

> So far, the only feedback about this idea has been negative, so at the
> moment, I am pretty convinced it isn't going to happen.  That doesn't mean
I
> won't try elsewhere :)

I would support bringing Poolman here. Although I'm far from an expert on
connection pools, I'm also willing to sign up as an initial committer to
help with that process. Just let me know what I can do to help.

--
Martin Cooper


>
> >
> > Yes jakarta developers should be able to work on what they want to work
> > on, but the founding projects of the commons was the pool and dbcp.
>
> Yep - certainly was my motivation.
>
> > Yes
> > it is possible to have multiple implementations of these.
>
> Yep.
>
> [SNIP]
> > Isn't the point of the
> > commons to not have multiple implementations of the same concept?
>
> Not necessarily...
>
>
>
> > "Geir Magnusson Jr." wrote:
> >>
> >> On 3/6/02 6:32 AM, "Ted Husted" <[EMAIL PROTECTED]> wrote:
> >>
> >>> The only point to anything at Jakarta is that the volunteers will work
> >>> on what they ~want~ to work on. If there are volunteers here who want
to
> >>> work on the dbcp, then we will continue to have a dbcp. If there are
> >>> volunteers here who want to bring Poolman here, then they could do
that
> >>> too.
> >>>
> >>> My only point is that the opportunity now exists; it's just a matter
of
> >>> whether anyone interested in working on connection pools is interested
> >>> in pursuing the Poolman codebase.
> >>
> >> I've talked to Sean every now and then about bringing Poolman here to
> >> Jakarta, and I have another query on this in to him now.
> >>
> >> I'm going to try and do it.
> >>
> >> geir
> >>
> >> --
> >> Geir Magnusson Jr.
[EMAIL PROTECTED]
> >> System and Software Consulting
> >> The bytecodes are language independent. - Sam Ruby
> >>
> >> --
> >> To unsubscribe, e-mail:
<mailto:[EMAIL PROTECTED]>
> >> For additional commands, e-mail:
<mailto:[EMAIL PROTECTED]>
> >
> > --
> > To unsubscribe, e-mail:
<mailto:[EMAIL PROTECTED]>
> > For additional commands, e-mail:
<mailto:[EMAIL PROTECTED]>
> >
>
> --
> Geir Magnusson Jr. [EMAIL PROTECTED]
> System and Software Consulting
> POC lives!
>
>
> --
> To unsubscribe, e-mail:
<mailto:[EMAIL PROTECTED]>
> For additional commands, e-mail:
<mailto:[EMAIL PROTECTED]>
>


--
To unsubscribe, e-mail:   <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>

Re: commons dbcp or pool problems

[Geir Magnusson Jr.]

On 3/6/02 2:53 PM, "John McNally" <[EMAIL PROTECTED]> wrote:

> 
> The code in sandbox/jdbc2pool uses the commons pool and DBCP's
> PreparedStatement pool while attempting to implement the jdbc2
> specification.  Geir and anyone else at jakarta who wants to work on a
> connection pool, why not join me in finishing this?  Geir, why are you
> saying you want to work on Poolman, what does it have that dbcp does
> not?

I am saying that I think Poolman is worth saving.  I rely on it in several
client projects, and I am sure there are many users of Poolman here, despite
the existence of the dbcp.

This is not a reflection on DBCP, but rather acting on the knowledge that
Poolman has a huge user base and a long track record of reliability.

So far, the only feedback about this idea has been negative, so at the
moment, I am pretty convinced it isn't going to happen.  That doesn't mean I
won't try elsewhere :)

> 
> Yes jakarta developers should be able to work on what they want to work
> on, but the founding projects of the commons was the pool and dbcp.

Yep - certainly was my motivation.

> Yes
> it is possible to have multiple implementations of these.

Yep.

[SNIP]
> Isn't the point of the
> commons to not have multiple implementations of the same concept?

Not necessarily...



> "Geir Magnusson Jr." wrote:
>> 
>> On 3/6/02 6:32 AM, "Ted Husted" <[EMAIL PROTECTED]> wrote:
>> 
>>> The only point to anything at Jakarta is that the volunteers will work
>>> on what they ~want~ to work on. If there are volunteers here who want to
>>> work on the dbcp, then we will continue to have a dbcp. If there are
>>> volunteers here who want to bring Poolman here, then they could do that
>>> too.
>>> 
>>> My only point is that the opportunity now exists; it's just a matter of
>>> whether anyone interested in working on connection pools is interested
>>> in pursuing the Poolman codebase.
>> 
>> I've talked to Sean every now and then about bringing Poolman here to
>> Jakarta, and I have another query on this in to him now.
>> 
>> I'm going to try and do it.
>> 
>> geir
>> 
>> --
>> Geir Magnusson Jr. [EMAIL PROTECTED]
>> System and Software Consulting
>> The bytecodes are language independent. - Sam Ruby
>> 
>> --
>> To unsubscribe, e-mail:   
>> For additional commands, e-mail: 
> 
> --
> To unsubscribe, e-mail:   
> For additional commands, e-mail: 
> 

-- 
Geir Magnusson Jr. [EMAIL PROTECTED]
System and Software Consulting
POC lives!


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[John McNally]

Why do you want to bring Poolman here?  Commons already has a pool and
connection pool.  In an attempt to allow jakarta-turbine-torque to use
any jdbc2 pool, I started looking for an open-source implementation and
found none.  Poolman and DBCP can be said to have the possibility of
presenting a DataSource front end, but neither attempt to implement the
specification fully.  As PreparedStatement pooling is not addressed
until jdbc3, you could say they are both ahead of the curve.

But pushing either of these without attempting to bring them up to at
least jdbc2 compliance does not make much sense.  I have been working on
this for over a month (intermittently, of course) and have received one
code submission from another developer, not a commons committer.  Craig
has submitted a jdbc2 style DataSource (meaning bean-like configuration)
example to the DBCP codebase that I think serves as a good example on
how to use the DBCP codebase in an easy to configure manner.  It does
not include much configurability and leaves out statement pooling,
however, and it still does not hook into the correct backend for a jdbc2
pool.

The code in sandbox/jdbc2pool uses the commons pool and DBCP's
PreparedStatement pool while attempting to implement the jdbc2
specification.  Geir and anyone else at jakarta who wants to work on a
connection pool, why not join me in finishing this?  Geir, why are you
saying you want to work on Poolman, what does it have that dbcp does
not?

Yes jakarta developers should be able to work on what they want to work
on, but the founding projects of the commons was the pool and dbcp.  Yes
it is possible to have multiple implementations of these.  We have an
object pool in turbine that we are still deciding on whether to keep it
maintained or dump it for the current commons pool.  We had our own dbcp
as well, which I do not think anyone wants to maintain as a separate
entity even though it works perfectly well.  Isn't the point of the
commons to not have multiple implementations of the same concept?  I
have to say I still do not quite understand what the commons is.

john mcnally

"Geir Magnusson Jr." wrote:
> 
> On 3/6/02 6:32 AM, "Ted Husted" <[EMAIL PROTECTED]> wrote:
> 
> > The only point to anything at Jakarta is that the volunteers will work
> > on what they ~want~ to work on. If there are volunteers here who want to
> > work on the dbcp, then we will continue to have a dbcp. If there are
> > volunteers here who want to bring Poolman here, then they could do that
> > too.
> >
> > My only point is that the opportunity now exists; it's just a matter of
> > whether anyone interested in working on connection pools is interested
> > in pursuing the Poolman codebase.
> 
> I've talked to Sean every now and then about bringing Poolman here to
> Jakarta, and I have another query on this in to him now.
> 
> I'm going to try and do it.
> 
> geir
> 
> --
> Geir Magnusson Jr. [EMAIL PROTECTED]
> System and Software Consulting
> The bytecodes are language independent. - Sam Ruby
> 
> --
> To unsubscribe, e-mail:   
> For additional commands, e-mail: 

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Bryan Field-Elliot]

What is the status of commons dbcp?

Apart from what look like minor edits from Craig, it's been 10 months
(according to CVS) since any major work has been done. Is this
considered done and production quality or it still in beta (or less?) 

I noticed that it's part of the Struts nightly build now.. Does that
vouch for its completeness in any way?

If dbcp is in good shape (which I'm trying to ask about), then is there
any point in also trying to bring poolman here?

Bryan

On Tue, 2002-03-05 at 21:00, Ted Husted wrote:

So in the event anyone is interested, the Poolman maintainer has
retired. 

"I no longer have time to maintain the PoolMan
JDBC driver and resource pooling and caching
 utility, and am now seeking a qualified,
comitted software engineer to assume control
over its future. If interested, please send email
to [EMAIL PROTECTED] with a brief summary
of your qualifications."

http://sourceforge.net/forum/forum.php?forum_id=44181

This really isn't my realm, but I thought I would bring it up in case
any one here would be up to working with Neville to bring poolman here.
There was some talk about that last year, so maybe it's time to revive
the discussions. Of course, we'd have to get the license changed, et
cetera.

Meanwhile, I'm tucking the last production release away for safekeeping. 

-Ted.

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Ted Husted]

So in the event anyone is interested, the Poolman maintainer has
retired. 

"I no longer have time to maintain the PoolMan
JDBC driver and resource pooling and caching
 utility, and am now seeking a qualified,
comitted software engineer to assume control
over its future. If interested, please send email
to [EMAIL PROTECTED] with a brief summary
of your qualifications."

http://sourceforge.net/forum/forum.php?forum_id=44181

This really isn't my realm, but I thought I would bring it up in case
any one here would be up to working with Neville to bring poolman here.
There was some talk about that last year, so maybe it's time to revive
the discussions. Of course, we'd have to get the license changed, et
cetera.

Meanwhile, I'm tucking the last production release away for safekeeping. 

-Ted.

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Juozas Baliuka]

> > > Craig
> > I see only one possible solution : to have two tagets in ant script, one
for
> > jdk1.3 and for 1.4,
> > conditional compling  is not very nice solution, but I see no choises.
> >
>
> Doing the conditional compilation part is pretty straightforward.  We'll
> also want to make the DBCP runtimes smart about which implementation class
> it uses for Connections based on what JVM the application is running in,
> without requiring the application itself to configure things any
> differently.
>
> Also, whoever builds a DBCP distribution bundle needs to make sure they
> run both compilers, so that both implementation classes are available in
> the JAR file ...
>
> Craig
 It is very trivial to implement Connection interface at runtime, but some
performance can be lost.
( see java.reflect.Proxy )


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

RE: commons dbcp or pool problems

[Aaron Smuts]

This isn't necessary.

> -Original Message-
> From: Craig R. McClanahan [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, February 26, 2002 12:41 PM
> To: Jakarta Commons Developers List
> Cc: [EMAIL PROTECTED]
> Subject: Re: commons dbcp or pool problems
> 
> 
> 
> On Tue, 26 Feb 2002, Juozas Baliuka wrote:
> 
> > Date: Tue, 26 Feb 2002 19:12:36 +0100
> > From: Juozas Baliuka <[EMAIL PROTECTED]>
> > Reply-To: Jakarta Commons Developers List  [EMAIL PROTECTED]>
> > To: Jakarta Commons Developers List
<[EMAIL PROTECTED]>,
> >  [EMAIL PROTECTED]
> > Subject: Re: commons dbcp or pool problems
> >
> > Hi,
> > >
> > > >
> > > > Does the pool you are working on work with JDK1.4 and 1.3?
> > > >
> > >
> > > This is a nasty issue, due to the changes in java.sql.Connection
in
> 1.4.
> > > Basically, it's not possible to write a single implementation of
> > > Connection that compiles under both 1.3 and 1.4.
> > >
> > > The current implementation compiles under 1.3, and runs under
either
> (as
> > > long as you don't try to call the new methods.  We'll need to come
up
> with
> > > a creative strategy to compile an alternative Connection
> implementation
> > > class under 1.4.
> > >
> > > > Aaron
> > >
> > > Craig
> > I see only one possible solution : to have two tagets in ant script,
one
> for
> > jdk1.3 and for 1.4,
> > conditional compling  is not very nice solution, but I see no
choises.
> >
> 
> Doing the conditional compilation part is pretty straightforward.
We'll
> also want to make the DBCP runtimes smart about which implementation
class
> it uses for Connections based on what JVM the application is running
in,
> without requiring the application itself to configure things any
> differently.
> 
> Also, whoever builds a DBCP distribution bundle needs to make sure
they
> run both compilers, so that both implementation classes are available
in
> the JAR file ...
> 
> Craig
> 
> 
> >
> > --
> > To unsubscribe, e-mail:   <mailto:commons-dev-
> [EMAIL PROTECTED]>
> > For additional commands, e-mail: <mailto:commons-dev-
> [EMAIL PROTECTED]>
> >
> >
> 
> 
> --
> To unsubscribe, e-mail:   <mailto:commons-dev-
> [EMAIL PROTECTED]>
> For additional commands, e-mail: <mailto:commons-dev-
> [EMAIL PROTECTED]>



--
To unsubscribe, e-mail:   <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>

RE: commons dbcp or pool problems

[Aaron Smuts]

I have a solution that will compile under both.  It is running in
production.  I'll have something to show in a day or so.

> -Original Message-
> From: Craig R. McClanahan [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, February 26, 2002 11:25 AM
> To: Jakarta Commons Developers List; [EMAIL PROTECTED]
> Subject: RE: commons dbcp or pool problems
> 
> 
> 
> On Tue, 26 Feb 2002, Aaron Smuts wrote:
> 
> >
> > Does the pool you are working on work with JDK1.4 and 1.3?
> >
> 
> This is a nasty issue, due to the changes in java.sql.Connection in
1.4.
> Basically, it's not possible to write a single implementation of
> Connection that compiles under both 1.3 and 1.4.
> 
> The current implementation compiles under 1.3, and runs under either
(as
> long as you don't try to call the new methods.  We'll need to come up
with
> a creative strategy to compile an alternative Connection
implementation
> class under 1.4.
> 
> > Aaron
> >
> 
> Craig



--
To unsubscribe, e-mail:   <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>

Re: commons dbcp or pool problems

[Craig R. McClanahan]

On Tue, 26 Feb 2002, Juozas Baliuka wrote:

> Date: Tue, 26 Feb 2002 19:12:36 +0100
> From: Juozas Baliuka <[EMAIL PROTECTED]>
> Reply-To: Jakarta Commons Developers List <[EMAIL PROTECTED]>
> To: Jakarta Commons Developers List <[EMAIL PROTECTED]>,
>      [EMAIL PROTECTED]
> Subject: Re: commons dbcp or pool problems
>
> Hi,
> >
> > >
> > > Does the pool you are working on work with JDK1.4 and 1.3?
> > >
> >
> > This is a nasty issue, due to the changes in java.sql.Connection in 1.4.
> > Basically, it's not possible to write a single implementation of
> > Connection that compiles under both 1.3 and 1.4.
> >
> > The current implementation compiles under 1.3, and runs under either (as
> > long as you don't try to call the new methods.  We'll need to come up with
> > a creative strategy to compile an alternative Connection implementation
> > class under 1.4.
> >
> > > Aaron
> >
> > Craig
> I see only one possible solution : to have two tagets in ant script, one for
> jdk1.3 and for 1.4,
> conditional compling  is not very nice solution, but I see no choises.
>

Doing the conditional compilation part is pretty straightforward.  We'll
also want to make the DBCP runtimes smart about which implementation class
it uses for Connections based on what JVM the application is running in,
without requiring the application itself to configure things any
differently.

Also, whoever builds a DBCP distribution bundle needs to make sure they
run both compilers, so that both implementation classes are available in
the JAR file ...

Craig


>
> --
> To unsubscribe, e-mail:   <mailto:[EMAIL PROTECTED]>
> For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
>
>


--
To unsubscribe, e-mail:   <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>

Re: commons dbcp or pool problems

[Juozas Baliuka]

Hi,
>
> >
> > Does the pool you are working on work with JDK1.4 and 1.3?
> >
>
> This is a nasty issue, due to the changes in java.sql.Connection in 1.4.
> Basically, it's not possible to write a single implementation of
> Connection that compiles under both 1.3 and 1.4.
>
> The current implementation compiles under 1.3, and runs under either (as
> long as you don't try to call the new methods.  We'll need to come up with
> a creative strategy to compile an alternative Connection implementation
> class under 1.4.
>
> > Aaron
>
> Craig
I see only one possible solution : to have two tagets in ant script, one for
jdk1.3 and for 1.4,
conditional compling  is not very nice solution, but I see no choises.


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

RE: commons dbcp or pool problems

[Aaron Smuts]

You can easily take out Oracle this way, even by just not closing
statements on reaped connections.  This is very dangerous.

I have a *practically* fool proof pooling technique that we are working
on.  I may have something in a few days.  To sort of give it away -- we
clean up after each request.  Guess how.  I may have already mentioned
it.

The programmers don't even have to release their resources after getting
a connection from the pool.  It isn't limited to servlets either.  A
friend came up with the technique.  I love it.

Even with a background reaper a single bad class can take out your
database under high load.  I've seen it happen.  It just isn't good
enough.

If your project gets large enough, you may never find all the possible
problem areas.

Does the pool you are working on work with JDK1.4 and 1.3?  

Aaron

> -Original Message-
> From: James Strachan [mailto:[EMAIL PROTECTED]]
> Sent: Monday, February 25, 2002 7:02 PM
> To: Jakarta Commons Developers List
> Subject: Re: commons dbcp or pool problems
> 
> Why not just create more connections when the pool is exhausted and
then
> log
> a warning message, so the bug can be tracked down later without
breaking
> production? Most databases can handle plenty of connections these days
> anyways.
> 
> James
> - Original Message -
> From: "Glenn Nielsen" <[EMAIL PROTECTED]>
> To: "Jakarta Commons Developers List" <[EMAIL PROTECTED]>
> Sent: Sunday, February 24, 2002 2:11 AM
> Subject: Re: commons dbcp or pool problems
> 
> 
> > I like this.  There is another option which could be added.
> >
> > clientTimeout
> >
> > It the pool exhausts its connections, it will review the list
> > of connections in use, the connection which exceeds the
clientTimeout
> > by the most gets recycled.  If all the Statements are tracked,
> > they can be closed (which closes any open result sets for those
> Statements).
> > And then the Connection can be closed.  A new connection would need
> > to be created to replace the one recycled, then the old connection
> > and Statement could get GC'd.
> >
> > This would help prevent those cases where a poorly written JSP page
> > or Servlet doesn't close and release its JDBC resources from
preventing
> > others from getting a JDBC connection.
> >
> > Logging of misbehaving JSP pages and servlets is important.
> >
> > I am all for adding the below patch and the option above.
> >
> > It sure would have saved me some time the last few days. :-)
> >
> > Regards,
> >
> > Glenn
> >
> > James House wrote:
> > >
> > > At 2/23/2002 01:25 PM -0600, Rodney Waldhoff wrote:
> > > > > A few months back I made my own hacks to
> > > > > DBCP in order to have it find places in
> > > > > our code that didn't free up DB resources
> > > > > properly.
> > > > >
> > > > > I was able to generate class names and
> > > > > line-numbers (stack trace) for every place
> > > > > in the code that a statement or result set
> > > > > was created but never closed, and also able
> > > > > to track where connections were borrowed
> > > > > and never returned.
> > > >
> > > > > If anyone's interested, I could try
> > > > > digging it up, and posting it.
> > > >
> > > >Sounds great, especially if we can do it relatively unobtrusively
(or
> > > >optionally).  By all means, put together a patch.
> > >
> > > I'll be happy to put together a patch.
> > >
> > > First let me know what the preference is of how it fits in:
> > >
> > > Background:
> > >
> > > The changes affect the following classes:
> > >
> > > PoolableConnection
> > > PoolableConnectionFactory
> > > PoolingDataSource
> > > DelegatingStatement
> > > DelegatingResultSet
> > > DelegatingPreparedStatement
> > > DelegatingCallableStatement
> > >
> > > The changes basically entail adding Lists to the objects to keep
track
> of
> > > the sub-objects that they've made, and also adding a member
variable
> that
> > > hold a reference to an Exception that is created (but not thrown)
at
> the
> > > time the Connection was borrowed or  at the time a Statement /
> ResultSet
> > > was made... so that it can be used to generate a stack trace that
> shows
> the
> > > code that borrowed/created the resource, when we detect 

RE: commons dbcp or pool problems

[James House]

At 2/23/2002 01:25 PM -0600, Rodney Waldhoff wrote:
>James House wrote:
>
> > A few months back I made my own hacks to
> > DBCP in order to have it find places in
> > our code that didn't free up DB resources
> > properly.



> > If anyone's interested, I could try
> > digging it up, and posting it.
>
>Sounds great, especially if we can do it relatively unobtrusively (or
>optionally).  By all means, put together a patch.

Well, I don't have any time soon to look at how to clean this thing up to 
make a reasonable patch.

I've posted my source for my changes here: 
http://www.interobjective.com/dbcpHacks.html

If it was a simple as doing the "diff" and creating the patch, I'd do it... 
But now that I've dug it up and looked at it (refreshed my memory about 
what I did) I think there are too many issues with my hacking - it is more 
obtrusive to existing code than I remembered it being, so I think there 
would end up being a lot of complaints about the patch -- and since I don't 
"own" DBCP I don't think I'm qualified to make any radical changes to it!

That said, please feel free to check it out. -- Anyone interested can take 
it and bend it into a reasonable patch, or simply steal any ideas you want, 
and make a good patch for DBCP.

James



--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

RE: commons dbcp or pool problems

[Aaron Smuts]

> 
> But in a production environment where I host the applications, not
write
> them, I don't want to be called at 2PM on a Sunday to reboot the app
> server
> if a customer's application stops working because their bad code
exhausted
> their db connection pool.
> 
> I really think this feature is important.  If you don't need it, don't
> enable it.
 
In my current project we have somewhere close to a million lines of
code.  Problems popup.  We made resource management magical.  We inspect
the returning thread of each request and make sure that it has released
its resources.  We can handle bad code (which does happen when you are
moving fast).  

We found that we had to have an option to turn off the background
connection reaper for standalone applications that need connections for
extended periods of time and we came up with another solution.  

 . . .

Aaron



--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Glenn Nielsen]

Comments intermixed below.

Juozas Baliuka wrote:
> 
> Hi,
> You can implement this without any extra thread. Pooling doesn't need
> threads, but amost all
>  pool and cashe  implementations use extra threads( it does nothing
> meaningful).
> Idea to use "timeout" is not very good, only crapy application needs this, I
> think it is
> good to have it for debug, but in this situation pool must fail and report
> error, not
> to try "find solution".
> 
> >
> > Actually, now that I'm more awake, I think the idea of a thread "timing
> > out" borrowed connections is a bad thing. -- It adds more code to DBCP,
> and
> > adds the overhead of an extra thread.  To top that off, its only purpose
> in
> > life is to work around code problems, rather than fixing them.
> >

Timing out and recovering borrowed connections doesn't have to require 
another thread.  Just recover a connection which has timed out by the
longest _if_ the pool of connections is exhausted.

> > With the other features there to show you exactly where you have holes in
> > your code, it's real easy to identify and fix them, while this feature
> just
> > discourages doing something about those holes.
> >

Logging those applications which do not release their resources is important.
And it would allow me to strongly encourage customers to fix their code
before it causes a problem.

But in a production environment where I host the applications, not write
them, I don't want to be called at 2PM on a Sunday to reboot the app server
if a customer's application stops working because their bad code exhausted
their db connection pool.

I really think this feature is important.  If you don't need it, don't
enable it.

> > James
> >
> > At 2/23/2002 08:40 PM -0700, you wrote:
> >
> > >Actually, I had it working that way for a while too, but had complaints
> > >(in-house) about the extra thread...  But I'd be happy to throw it in.
> > >
> > >I guess the question is:  Should this be in the standard DBCP stuff (with
> > >an on-off switch) or as separate classes only for use during debugging?
> > >
> > >James
> > >
> > >At 2/23/2002 08:11 PM -0600, you wrote:
> > >>I like this.  There is another option which could be added.
> > >>
> > >>clientTimeout
> > >>
> > >>It the pool exhausts its connections, it will review the list
> > >>of connections in use, the connection which exceeds the clientTimeout
> > >>by the most gets recycled.  If all the Statements are tracked,
> > >>they can be closed (which closes any open result sets for those
> Statements).
> > >>And then the Connection can be closed.  A new connection would need
> > >>to be created to replace the one recycled, then the old connection
> > >>and Statement could get GC'd.
> > >>
> > >>This would help prevent those cases where a poorly written JSP page
> > >>or Servlet doesn't close and release its JDBC resources from preventing
> > >>others from getting a JDBC connection.
> > >>
> > >>Logging of misbehaving JSP pages and servlets is important.
> > >>
> > >>I am all for adding the below patch and the option above.
> > >>
> > >>It sure would have saved me some time the last few days. :-)
> > >>
> > >>Regards,
> > >>
> > >>Glenn
> > >>
> > >>James House wrote:
> > >> >
> > >> > At 2/23/2002 01:25 PM -0600, Rodney Waldhoff wrote:
> > >> > > > A few months back I made my own hacks to
> > >> > > > DBCP in order to have it find places in
> > >> > > > our code that didn't free up DB resources
> > >> > > > properly.
> > >> > > >
> > >> > > > I was able to generate class names and
> > >> > > > line-numbers (stack trace) for every place
> > >> > > > in the code that a statement or result set
> > >> > > > was created but never closed, and also able
> > >> > > > to track where connections were borrowed
> > >> > > > and never returned.
> > >> > >
> > >> > > > If anyone's interested, I could try
> > >> > > > digging it up, and posting it.
> > >> > >
> > >> > >Sounds great, especially if we can do it relatively unobtrusively
> (or
> > >> > >optionally).  By all means, put together a patch.
> > >> >
> > >> > I'll be happy to put together a patch.
> > >> >
> > >> > First let me know what the preference is of how it fits in:
> > >> >
> > >> > Background:
> > >> >
> > >> > The changes affect the following classes:
> > >> >
> > >> > PoolableConnection
> > >> > PoolableConnectionFactory
> > >> > PoolingDataSource
> > >> > DelegatingStatement
> > >> > DelegatingResultSet
> > >> > DelegatingPreparedStatement
> > >> > DelegatingCallableStatement
> > >> >
> > >> > The changes basically entail adding Lists to the objects to keep
> track of
> > >> > the sub-objects that they've made, and also adding a member variable
> that
> > >> > hold a reference to an Exception that is created (but not thrown) at
> the
> > >> > time the Connection was borrowed or  at the time a Statement /
> ResultSet
> > >> > was made... so that it can be used to generate a stack trace that
> > >> shows the
> > >> > code that borrowed/created the resource, when we

Re: commons dbcp or pool problems

[Juozas Baliuka]

Hi,
You can implement this without any extra thread. Pooling doesn't need
threads, but amost all
 pool and cashe  implementations use extra threads( it does nothing
meaningful).
Idea to use "timeout" is not very good, only crapy application needs this, I
think it is
good to have it for debug, but in this situation pool must fail and report
error, not
to try "find solution".

>
> Actually, now that I'm more awake, I think the idea of a thread "timing
> out" borrowed connections is a bad thing. -- It adds more code to DBCP,
and
> adds the overhead of an extra thread.  To top that off, its only purpose
in
> life is to work around code problems, rather than fixing them.
>
> With the other features there to show you exactly where you have holes in
> your code, it's real easy to identify and fix them, while this feature
just
> discourages doing something about those holes.
>
> James
>
> At 2/23/2002 08:40 PM -0700, you wrote:
>
> >Actually, I had it working that way for a while too, but had complaints
> >(in-house) about the extra thread...  But I'd be happy to throw it in.
> >
> >I guess the question is:  Should this be in the standard DBCP stuff (with
> >an on-off switch) or as separate classes only for use during debugging?
> >
> >James
> >
> >At 2/23/2002 08:11 PM -0600, you wrote:
> >>I like this.  There is another option which could be added.
> >>
> >>clientTimeout
> >>
> >>It the pool exhausts its connections, it will review the list
> >>of connections in use, the connection which exceeds the clientTimeout
> >>by the most gets recycled.  If all the Statements are tracked,
> >>they can be closed (which closes any open result sets for those
Statements).
> >>And then the Connection can be closed.  A new connection would need
> >>to be created to replace the one recycled, then the old connection
> >>and Statement could get GC'd.
> >>
> >>This would help prevent those cases where a poorly written JSP page
> >>or Servlet doesn't close and release its JDBC resources from preventing
> >>others from getting a JDBC connection.
> >>
> >>Logging of misbehaving JSP pages and servlets is important.
> >>
> >>I am all for adding the below patch and the option above.
> >>
> >>It sure would have saved me some time the last few days. :-)
> >>
> >>Regards,
> >>
> >>Glenn
> >>
> >>James House wrote:
> >> >
> >> > At 2/23/2002 01:25 PM -0600, Rodney Waldhoff wrote:
> >> > > > A few months back I made my own hacks to
> >> > > > DBCP in order to have it find places in
> >> > > > our code that didn't free up DB resources
> >> > > > properly.
> >> > > >
> >> > > > I was able to generate class names and
> >> > > > line-numbers (stack trace) for every place
> >> > > > in the code that a statement or result set
> >> > > > was created but never closed, and also able
> >> > > > to track where connections were borrowed
> >> > > > and never returned.
> >> > >
> >> > > > If anyone's interested, I could try
> >> > > > digging it up, and posting it.
> >> > >
> >> > >Sounds great, especially if we can do it relatively unobtrusively
(or
> >> > >optionally).  By all means, put together a patch.
> >> >
> >> > I'll be happy to put together a patch.
> >> >
> >> > First let me know what the preference is of how it fits in:
> >> >
> >> > Background:
> >> >
> >> > The changes affect the following classes:
> >> >
> >> > PoolableConnection
> >> > PoolableConnectionFactory
> >> > PoolingDataSource
> >> > DelegatingStatement
> >> > DelegatingResultSet
> >> > DelegatingPreparedStatement
> >> > DelegatingCallableStatement
> >> >
> >> > The changes basically entail adding Lists to the objects to keep
track of
> >> > the sub-objects that they've made, and also adding a member variable
that
> >> > hold a reference to an Exception that is created (but not thrown) at
the
> >> > time the Connection was borrowed or  at the time a Statement /
ResultSet
> >> > was made... so that it can be used to generate a stack trace that
> >> shows the
> >> > code that borrowed/created the resource, when we detect that it
wasn't
> >> > closed/returned properly.  This detection is done by removing the
objects
> >> > from the before mentioned lists as close() is called on them, and
then
> >> > checking if there are still objects in the lists when the connection
is
> >> > returned to the pool, or by calling a new method "detectLeaks()" on
> >> > PoolingDataSource to find un-returned connections.
> >> >
> >> > So you can see the changes aren't extremely intrusive (as far as
changing
> >> > existing code) but do ADD roughly 10 - 35  lines of code to each of
the
> >> > classes listed above.
> >> >
> >> > Option One:
> >> >
> >> > Create a patch that alters these classes, and allows you to turn
> >> on/off the
> >> > debugging features at the time you create your instance of
> >> > PoolableConnectionFactory.
> >> >
> >> > Option Two:
> >> >
> >> > Create new versions of these classes that have the feature always on,
and
> >> > you simply use a class named something like
> >> > "Deb

Re: commons dbcp or pool problems

[James House]

Actually, now that I'm more awake, I think the idea of a thread "timing 
out" borrowed connections is a bad thing. -- It adds more code to DBCP, and 
adds the overhead of an extra thread.  To top that off, its only purpose in 
life is to work around code problems, rather than fixing them.

With the other features there to show you exactly where you have holes in 
your code, it's real easy to identify and fix them, while this feature just 
discourages doing something about those holes.

James

At 2/23/2002 08:40 PM -0700, you wrote:

>Actually, I had it working that way for a while too, but had complaints 
>(in-house) about the extra thread...  But I'd be happy to throw it in.
>
>I guess the question is:  Should this be in the standard DBCP stuff (with 
>an on-off switch) or as separate classes only for use during debugging?
>
>James
>
>At 2/23/2002 08:11 PM -0600, you wrote:
>>I like this.  There is another option which could be added.
>>
>>clientTimeout
>>
>>It the pool exhausts its connections, it will review the list
>>of connections in use, the connection which exceeds the clientTimeout
>>by the most gets recycled.  If all the Statements are tracked,
>>they can be closed (which closes any open result sets for those Statements).
>>And then the Connection can be closed.  A new connection would need
>>to be created to replace the one recycled, then the old connection
>>and Statement could get GC'd.
>>
>>This would help prevent those cases where a poorly written JSP page
>>or Servlet doesn't close and release its JDBC resources from preventing
>>others from getting a JDBC connection.
>>
>>Logging of misbehaving JSP pages and servlets is important.
>>
>>I am all for adding the below patch and the option above.
>>
>>It sure would have saved me some time the last few days. :-)
>>
>>Regards,
>>
>>Glenn
>>
>>James House wrote:
>> >
>> > At 2/23/2002 01:25 PM -0600, Rodney Waldhoff wrote:
>> > > > A few months back I made my own hacks to
>> > > > DBCP in order to have it find places in
>> > > > our code that didn't free up DB resources
>> > > > properly.
>> > > >
>> > > > I was able to generate class names and
>> > > > line-numbers (stack trace) for every place
>> > > > in the code that a statement or result set
>> > > > was created but never closed, and also able
>> > > > to track where connections were borrowed
>> > > > and never returned.
>> > >
>> > > > If anyone's interested, I could try
>> > > > digging it up, and posting it.
>> > >
>> > >Sounds great, especially if we can do it relatively unobtrusively (or
>> > >optionally).  By all means, put together a patch.
>> >
>> > I'll be happy to put together a patch.
>> >
>> > First let me know what the preference is of how it fits in:
>> >
>> > Background:
>> >
>> > The changes affect the following classes:
>> >
>> > PoolableConnection
>> > PoolableConnectionFactory
>> > PoolingDataSource
>> > DelegatingStatement
>> > DelegatingResultSet
>> > DelegatingPreparedStatement
>> > DelegatingCallableStatement
>> >
>> > The changes basically entail adding Lists to the objects to keep track of
>> > the sub-objects that they've made, and also adding a member variable that
>> > hold a reference to an Exception that is created (but not thrown) at the
>> > time the Connection was borrowed or  at the time a Statement / ResultSet
>> > was made... so that it can be used to generate a stack trace that 
>> shows the
>> > code that borrowed/created the resource, when we detect that it wasn't
>> > closed/returned properly.  This detection is done by removing the objects
>> > from the before mentioned lists as close() is called on them, and then
>> > checking if there are still objects in the lists when the connection is
>> > returned to the pool, or by calling a new method "detectLeaks()" on
>> > PoolingDataSource to find un-returned connections.
>> >
>> > So you can see the changes aren't extremely intrusive (as far as changing
>> > existing code) but do ADD roughly 10 - 35  lines of code to each of the
>> > classes listed above.
>> >
>> > Option One:
>> >
>> > Create a patch that alters these classes, and allows you to turn 
>> on/off the
>> > debugging features at the time you create your instance of
>> > PoolableConnectionFactory.
>> >
>> > Option Two:
>> >
>> > Create new versions of these classes that have the feature always on, and
>> > you simply use a class named something like
>> > "DebugPoolableConnectionFactory" instead of "PoolableConnectionFactory" as
>> > you create your datasource.
>> >
>> > Also, I submitted a couple suggested patches back on January 17th (2
>> > separate e-mails to this mail list with the subject starting with [DBCP]),
>> > but never heard any response - albeit I didn't submit formal patches.
>> >
>> > I'd be happy to create these patches at the same time if you agree 
>> with them.
>> >
>> > James
>> >
>> > --
>> > To unsubscribe, 
>> e-mail:   
>> > For additional commands, e-mail: 
>> 
>>
>>

Re: commons dbcp or pool problems

[James House]

Actually, I had it working that way for a while too, but had complaints 
(in-house) about the extra thread...  But I'd be happy to throw it in.

I guess the question is:  Should this be in the standard DBCP stuff (with 
an on-off switch) or as separate classes only for use during debugging?

James

At 2/23/2002 08:11 PM -0600, you wrote:
>I like this.  There is another option which could be added.
>
>clientTimeout
>
>It the pool exhausts its connections, it will review the list
>of connections in use, the connection which exceeds the clientTimeout
>by the most gets recycled.  If all the Statements are tracked,
>they can be closed (which closes any open result sets for those Statements).
>And then the Connection can be closed.  A new connection would need
>to be created to replace the one recycled, then the old connection
>and Statement could get GC'd.
>
>This would help prevent those cases where a poorly written JSP page
>or Servlet doesn't close and release its JDBC resources from preventing
>others from getting a JDBC connection.
>
>Logging of misbehaving JSP pages and servlets is important.
>
>I am all for adding the below patch and the option above.
>
>It sure would have saved me some time the last few days. :-)
>
>Regards,
>
>Glenn
>
>James House wrote:
> >
> > At 2/23/2002 01:25 PM -0600, Rodney Waldhoff wrote:
> > > > A few months back I made my own hacks to
> > > > DBCP in order to have it find places in
> > > > our code that didn't free up DB resources
> > > > properly.
> > > >
> > > > I was able to generate class names and
> > > > line-numbers (stack trace) for every place
> > > > in the code that a statement or result set
> > > > was created but never closed, and also able
> > > > to track where connections were borrowed
> > > > and never returned.
> > >
> > > > If anyone's interested, I could try
> > > > digging it up, and posting it.
> > >
> > >Sounds great, especially if we can do it relatively unobtrusively (or
> > >optionally).  By all means, put together a patch.
> >
> > I'll be happy to put together a patch.
> >
> > First let me know what the preference is of how it fits in:
> >
> > Background:
> >
> > The changes affect the following classes:
> >
> > PoolableConnection
> > PoolableConnectionFactory
> > PoolingDataSource
> > DelegatingStatement
> > DelegatingResultSet
> > DelegatingPreparedStatement
> > DelegatingCallableStatement
> >
> > The changes basically entail adding Lists to the objects to keep track of
> > the sub-objects that they've made, and also adding a member variable that
> > hold a reference to an Exception that is created (but not thrown) at the
> > time the Connection was borrowed or  at the time a Statement / ResultSet
> > was made... so that it can be used to generate a stack trace that shows the
> > code that borrowed/created the resource, when we detect that it wasn't
> > closed/returned properly.  This detection is done by removing the objects
> > from the before mentioned lists as close() is called on them, and then
> > checking if there are still objects in the lists when the connection is
> > returned to the pool, or by calling a new method "detectLeaks()" on
> > PoolingDataSource to find un-returned connections.
> >
> > So you can see the changes aren't extremely intrusive (as far as changing
> > existing code) but do ADD roughly 10 - 35  lines of code to each of the
> > classes listed above.
> >
> > Option One:
> >
> > Create a patch that alters these classes, and allows you to turn on/off the
> > debugging features at the time you create your instance of
> > PoolableConnectionFactory.
> >
> > Option Two:
> >
> > Create new versions of these classes that have the feature always on, and
> > you simply use a class named something like
> > "DebugPoolableConnectionFactory" instead of "PoolableConnectionFactory" as
> > you create your datasource.
> >
> > Also, I submitted a couple suggested patches back on January 17th (2
> > separate e-mails to this mail list with the subject starting with [DBCP]),
> > but never heard any response - albeit I didn't submit formal patches.
> >
> > I'd be happy to create these patches at the same time if you agree with 
> them.
> >
> > James
> >
> > --
> > To unsubscribe, 
> e-mail:   
> > For additional commands, e-mail: 
> 
>
>--
>--
>Glenn Nielsen [EMAIL PROTECTED] | /* Spelin donut madder|
>MOREnet System Programming   |  * if iz ina coment.  |
>Missouri Research and Education Network  |  */   |
>--
>
>--
>To unsubscribe, e-mail:   
>For additional commands, e-mail: 


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Glenn Nielsen]

I like this.  There is another option which could be added.

clientTimeout

It the pool exhausts its connections, it will review the list
of connections in use, the connection which exceeds the clientTimeout
by the most gets recycled.  If all the Statements are tracked,
they can be closed (which closes any open result sets for those Statements).
And then the Connection can be closed.  A new connection would need
to be created to replace the one recycled, then the old connection
and Statement could get GC'd.

This would help prevent those cases where a poorly written JSP page
or Servlet doesn't close and release its JDBC resources from preventing
others from getting a JDBC connection.

Logging of misbehaving JSP pages and servlets is important.

I am all for adding the below patch and the option above.

It sure would have saved me some time the last few days. :-)

Regards,

Glenn

James House wrote:
> 
> At 2/23/2002 01:25 PM -0600, Rodney Waldhoff wrote:
> > > A few months back I made my own hacks to
> > > DBCP in order to have it find places in
> > > our code that didn't free up DB resources
> > > properly.
> > >
> > > I was able to generate class names and
> > > line-numbers (stack trace) for every place
> > > in the code that a statement or result set
> > > was created but never closed, and also able
> > > to track where connections were borrowed
> > > and never returned.
> >
> > > If anyone's interested, I could try
> > > digging it up, and posting it.
> >
> >Sounds great, especially if we can do it relatively unobtrusively (or
> >optionally).  By all means, put together a patch.
> 
> I'll be happy to put together a patch.
> 
> First let me know what the preference is of how it fits in:
> 
> Background:
> 
> The changes affect the following classes:
> 
> PoolableConnection
> PoolableConnectionFactory
> PoolingDataSource
> DelegatingStatement
> DelegatingResultSet
> DelegatingPreparedStatement
> DelegatingCallableStatement
> 
> The changes basically entail adding Lists to the objects to keep track of
> the sub-objects that they've made, and also adding a member variable that
> hold a reference to an Exception that is created (but not thrown) at the
> time the Connection was borrowed or  at the time a Statement / ResultSet
> was made... so that it can be used to generate a stack trace that shows the
> code that borrowed/created the resource, when we detect that it wasn't
> closed/returned properly.  This detection is done by removing the objects
> from the before mentioned lists as close() is called on them, and then
> checking if there are still objects in the lists when the connection is
> returned to the pool, or by calling a new method "detectLeaks()" on
> PoolingDataSource to find un-returned connections.
> 
> So you can see the changes aren't extremely intrusive (as far as changing
> existing code) but do ADD roughly 10 - 35  lines of code to each of the
> classes listed above.
> 
> Option One:
> 
> Create a patch that alters these classes, and allows you to turn on/off the
> debugging features at the time you create your instance of
> PoolableConnectionFactory.
> 
> Option Two:
> 
> Create new versions of these classes that have the feature always on, and
> you simply use a class named something like
> "DebugPoolableConnectionFactory" instead of "PoolableConnectionFactory" as
> you create your datasource.
> 
> Also, I submitted a couple suggested patches back on January 17th (2
> separate e-mails to this mail list with the subject starting with [DBCP]),
> but never heard any response - albeit I didn't submit formal patches.
> 
> I'd be happy to create these patches at the same time if you agree with them.
> 
> James
> 
> --
> To unsubscribe, e-mail:   
> For additional commands, e-mail: 

-- 
--
Glenn Nielsen [EMAIL PROTECTED] | /* Spelin donut madder|
MOREnet System Programming   |  * if iz ina coment.  |
Missouri Research and Education Network  |  */   |
--

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

RE: commons dbcp or pool problems

[James House]

At 2/23/2002 01:25 PM -0600, Rodney Waldhoff wrote:
> > A few months back I made my own hacks to
> > DBCP in order to have it find places in
> > our code that didn't free up DB resources
> > properly.
> >
> > I was able to generate class names and
> > line-numbers (stack trace) for every place
> > in the code that a statement or result set
> > was created but never closed, and also able
> > to track where connections were borrowed
> > and never returned.
>
> > If anyone's interested, I could try
> > digging it up, and posting it.
>
>Sounds great, especially if we can do it relatively unobtrusively (or
>optionally).  By all means, put together a patch.

I'll be happy to put together a patch.

First let me know what the preference is of how it fits in:

Background:

The changes affect the following classes:

PoolableConnection
PoolableConnectionFactory
PoolingDataSource
DelegatingStatement
DelegatingResultSet
DelegatingPreparedStatement
DelegatingCallableStatement

The changes basically entail adding Lists to the objects to keep track of 
the sub-objects that they've made, and also adding a member variable that 
hold a reference to an Exception that is created (but not thrown) at the 
time the Connection was borrowed or  at the time a Statement / ResultSet 
was made... so that it can be used to generate a stack trace that shows the 
code that borrowed/created the resource, when we detect that it wasn't 
closed/returned properly.  This detection is done by removing the objects 
from the before mentioned lists as close() is called on them, and then 
checking if there are still objects in the lists when the connection is 
returned to the pool, or by calling a new method "detectLeaks()" on 
PoolingDataSource to find un-returned connections.

So you can see the changes aren't extremely intrusive (as far as changing 
existing code) but do ADD roughly 10 - 35  lines of code to each of the 
classes listed above.

Option One:

Create a patch that alters these classes, and allows you to turn on/off the 
debugging features at the time you create your instance of 
PoolableConnectionFactory.

Option Two:

Create new versions of these classes that have the feature always on, and 
you simply use a class named something like 
"DebugPoolableConnectionFactory" instead of "PoolableConnectionFactory" as 
you create your datasource.



Also, I submitted a couple suggested patches back on January 17th (2 
separate e-mails to this mail list with the subject starting with [DBCP]), 
but never heard any response - albeit I didn't submit formal patches.

I'd be happy to create these patches at the same time if you agree with them.

James



--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

RE: commons dbcp or pool problems

[Waldhoff, Rodney]

James House wrote:

> A few months back I made my own hacks to 
> DBCP in order to have it find places in 
> our code that didn't free up DB resources 
> properly.
>
> I was able to generate class names and 
> line-numbers (stack trace) for every place 
> in the code that a statement or result set 
> was created but never closed, and also able 
> to track where connections were borrowed 
> and never returned.

> If anyone's interested, I could try 
> digging it up, and posting it.

Sounds great, especially if we can do it relatively unobtrusively (or
optionally).  By all means, put together a patch.

RE: commons dbcp or pool problems

[Gerhard Froehlich]

Hi,

>> >> Worse than using sql within a JSP, the customer turned almost their
>> >> entire web site into a bunch of JSP pages to dynamically generate
>> >> the content from a db.  But they only change the db 5-6 times a day.
>> >> So their pages generate all this load and are open to all kinds of
>> >> potential failures for what is essentially static content.  I strongly
>> >> recommended to them that they switch it over to a publish system where
>> >> they make changes to the db then generate the static html pages.
>> >> But of course the customer says they don't have time to do that,
>> >> they have something that "works".  This is for a site with 500,000
>> >> page views per month with spikes of 6-10k page views per hour.
>> >
>> >
>> >yesterday, just out of curiosity I conducted some simple benchmark test
>> >which measured how many pages per second can a client retrieve.
>> >
>> >the client was a simple java program which requested pages from a given
>url
>> >in a loop and then calculated the numbers. all network delays were
>eliminated
>> >just by running it on the same computer, which is PII 350, running win
>NT.
>> >
>> >first I gave it an url of jsp page served by tomcat 3.2. the maximum I
>got was 90 pages/sec
>> >then I gave it an url of jsp page served by resin: 196 pages/sec
>> >on tomcat 4.0 it was: 189 pages/second
>> >then I gave it an url of php page: 95 pages/sec
>> >and finally I tested static html page served by apache: 208 pages/sec
>> >
>> >of course this is not a real test, but however I got an impression that
>> >the difference between static page and jsp (in some good container) is
>not that
>> >big (of course if you don't do sql query from jsp).
>>
>> Maybe, but it's more about  "how do I design a proper web-appl" and SoC
>> -> to do easy maintainance and enhancements in the future
>>
>>   ~Gerhard
>Yes, it very possile to implement logics in JSP, put jdbc code, RMI ... and
>use cache for this
>stuff, It will be very fast to implement and you will have a very good
>performance, but
>It not solution, It is crap, you will need to remove you "fast application"
>later.

A collegue of mine is in the moment the tech. leader in a project. His devs
are some external guy's. This guy's implemented the whole appl. in JSP such
as BusinessLogic and SQL statements. You should hear him swearing. Ok with
the new IBM Websphere Studio Application Developer (now you know where I'm 
working) it's possible to "compile" the JSP's to check at least Syntax (I'm
sure there are some other tools around for pre-checking JSPs, but I don't use 
JSPs). But this guys doing all in some editor and they see their mistakes only,
when they deployed the web application. That's crap or, ignorant and a waste
of time.

  ~Gerhard


Give me ambiguity or give me something else.


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

RE: commons dbcp or pool problems

[Gerhard Froehlich]

Hi,

>> Worse than using sql within a JSP, the customer turned almost their
>> entire web site into a bunch of JSP pages to dynamically generate
>> the content from a db.  But they only change the db 5-6 times a day.
>> So their pages generate all this load and are open to all kinds of
>> potential failures for what is essentially static content.  I strongly
>> recommended to them that they switch it over to a publish system where
>> they make changes to the db then generate the static html pages.
>> But of course the customer says they don't have time to do that,
>> they have something that "works".  This is for a site with 500,000
>> page views per month with spikes of 6-10k page views per hour.
>
>
>yesterday, just out of curiosity I conducted some simple benchmark test
>which measured how many pages per second can a client retrieve.
>
>the client was a simple java program which requested pages from a given url
>in a loop and then calculated the numbers. all network delays were eliminated
>just by running it on the same computer, which is PII 350, running win NT.
>
>first I gave it an url of jsp page served by tomcat 3.2. the maximum I got was 90 
>pages/sec
>then I gave it an url of jsp page served by resin: 196 pages/sec
>on tomcat 4.0 it was: 189 pages/second
>then I gave it an url of php page: 95 pages/sec
>and finally I tested static html page served by apache: 208 pages/sec
>
>of course this is not a real test, but however I got an impression that
>the difference between static page and jsp (in some good container) is not that
>big (of course if you don't do sql query from jsp).

Maybe, but it's more about  "how do I design a proper web-appl" and SoC 
-> to do easy maintainance and enhancements in the future

  ~Gerhard
 

You can't fall off the floor



--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: OT: Helping Glenn (was Re: commons dbcp or pool problems)

[Glenn Nielsen]

Thanks for the suggestion.

The issue of caching pages was discussed weeks ago with the customer.
But of course the customer doesn't have time to make changes now,
it has to go into production.  They have something that "works".

If they eventually have time to redesign the application I will
recommend that they change the application to regenerate and publish
static pages based on changes to the data rather than implement caching.

Glenn

Lavandowska wrote:
> 
> --- Glenn Nielsen <[EMAIL PROTECTED]> wrote:
> > Worse than using sql within a JSP, the customer turned almost their
> > entire web site into a bunch of JSP pages to dynamically generate
> > the content from a db.  But they only change the db 5-6 times a day.
> 
> In the Commons-Pool is a taglib for caching portions (or all) of a
> page.  Since they are already using this package (for DBCP) perhaps it
> wouldn't be hard to convince them to cache the pages... ?
> 
> Lance
> 
> __
> Do You Yahoo!?
> Yahoo! Sports - Coverage of the 2002 Olympic Games
> http://sports.yahoo.com
> 
> --
> To unsubscribe, e-mail:   
> For additional commands, e-mail: 

-- 
--
Glenn Nielsen [EMAIL PROTECTED] | /* Spelin donut madder|
MOREnet System Programming   |  * if iz ina coment.  |
Missouri Research and Education Network  |  */   |
--

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

OT: Helping Glenn (was Re: commons dbcp or pool problems)

[Lavandowska]

--- Glenn Nielsen <[EMAIL PROTECTED]> wrote:
> Worse than using sql within a JSP, the customer turned almost their
> entire web site into a bunch of JSP pages to dynamically generate
> the content from a db.  But they only change the db 5-6 times a day.

In the Commons-Pool is a taglib for caching portions (or all) of a
page.  Since they are already using this package (for DBCP) perhaps it
wouldn't be hard to convince them to cache the pages... ?

Lance

__
Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games
http://sports.yahoo.com

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Dmitry Skavish]

> Worse than using sql within a JSP, the customer turned almost their
> entire web site into a bunch of JSP pages to dynamically generate
> the content from a db.  But they only change the db 5-6 times a day.
> So their pages generate all this load and are open to all kinds of
> potential failures for what is essentially static content.  I strongly
> recommended to them that they switch it over to a publish system where
> they make changes to the db then generate the static html pages.
> But of course the customer says they don't have time to do that,
> they have something that "works".  This is for a site with 500,000
> page views per month with spikes of 6-10k page views per hour.


yesterday, just out of curiosity I conducted some simple benchmark test
which measured how many pages per second can a client retrieve.

the client was a simple java program which requested pages from a given url
in a loop and then calculated the numbers. all network delays were eliminated
just by running it on the same computer, which is PII 350, running win NT.

first I gave it an url of jsp page served by tomcat 3.2. the maximum I got was 90 
pages/sec
then I gave it an url of jsp page served by resin: 196 pages/sec
on tomcat 4.0 it was: 189 pages/second
then I gave it an url of php page: 95 pages/sec
and finally I tested static html page served by apache: 208 pages/sec

of course this is not a real test, but however I got an impression that
the difference between static page and jsp (in some good container) is not that
big (of course if you don't do sql query from jsp).

-- 
Dmitry Skavish
---
Boston, MA, USA
tel. +1 781 370-6909
http://www.flashgap.com


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

RE: commons dbcp or pool problems

[Gerhard Froehlich]

Hi,

>> >I have reviewed the customers JSP pages and found one page
>> >where they were not wrapping their use of a db connection within a
>> >try {} block.  I have informed them how to fix it.
>> >
>> >But this one page does not explain running out of connections in a pool
>> >with a max of 75 connections.  In the logs that page only failed twice
>> >during the run of Tomcat before it ran out of db connections.
>> 
>> I know deadline is coming closer, customer is in a bad mood and the
>> requirements are doubled during the project, but I recommend to avoid
>> any SQL and Busines Logic in JSP. They are hard to debug and somehow
>> that's dirty. IMHO JSP it's for presentation.
>> 
>
>Yeah, I can't argue with that.  The problem is the customer wrote and
>or contracted out work to develop the application.  I just have to worry
>about hosting it.
>
>Worse than using sql within a JSP, the customer turned almost their
>entire web site into a bunch of JSP pages to dynamically generate
>the content from a db.  But they only change the db 5-6 times a day.
>So their pages generate all this load and are open to all kinds of
>potential failures for what is essentially static content.  I strongly
>recommended to them that they switch it over to a publish system where
>they make changes to the db then generate the static html pages.
>But of course the customer says they don't have time to do that,
>they have something that "works".  This is for a site with 500,000
>page views per month with spikes of 6-10k page views per hour.
>
>Sigh...

Dude, I can feel you pain, believe me...

  ~Gerhard
 
--
Hey! It compiles! Ship it!
--

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[James House]

At 2/22/2002 08:47 PM +0100, Juozas Baliuka wrote:
>It is very dificult to find pooling related bugs  in application.
>May be it has meaning to implement something like "DebugPool" to help find
>problems
>in users code.
>I can think about this, if this idea is interesting.

A few months back I made my own hacks to DBCP in order to have it find 
places in our code that didn't free up DB resources properly.

I was able to generate class names and line-numbers (stack trace) for every 
place in the code that a statement or result set was created but never 
closed, and also able to track where connections were borrowed and never 
returned.

All of this was pretty easy to do (an hour's work or so), and only required 
mods to the dbcp classes.

If anyone's interested, I could try digging it up, and posting it.

James


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Glenn Nielsen]

Gerhard Froehlich wrote:
> 
> Hi,
> 
> >I have reviewed the customers JSP pages and found one page
> >where they were not wrapping their use of a db connection within a
> >try {} block.  I have informed them how to fix it.
> >
> >But this one page does not explain running out of connections in a pool
> >with a max of 75 connections.  In the logs that page only failed twice
> >during the run of Tomcat before it ran out of db connections.
> 
> I know deadline is coming closer, customer is in a bad mood and the
> requirements are doubled during the project, but I recommend to avoid
> any SQL and Busines Logic in JSP. They are hard to debug and somehow
> that's dirty. IMHO JSP it's for presentation.
> 

Yeah, I can't argue with that.  The problem is the customer wrote and
or contracted out work to develop the application.  I just have to worry
about hosting it.

Worse than using sql within a JSP, the customer turned almost their
entire web site into a bunch of JSP pages to dynamically generate
the content from a db.  But they only change the db 5-6 times a day.
So their pages generate all this load and are open to all kinds of
potential failures for what is essentially static content.  I strongly
recommended to them that they switch it over to a publish system where
they make changes to the db then generate the static html pages.
But of course the customer says they don't have time to do that,
they have something that "works".  This is for a site with 500,000
page views per month with spikes of 6-10k page views per hour.

Sigh...

Glenn

--
Glenn Nielsen [EMAIL PROTECTED] | /* Spelin donut madder|
MOREnet System Programming   |  * if iz ina coment.  |
Missouri Research and Education Network  |  */   |
--

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

RE: commons dbcp or pool problems

[Gerhard Froehlich]

Hi,
 
>I have reviewed the customers JSP pages and found one page
>where they were not wrapping their use of a db connection within a
>try {} block.  I have informed them how to fix it.
>
>But this one page does not explain running out of connections in a pool
>with a max of 75 connections.  In the logs that page only failed twice
>during the run of Tomcat before it ran out of db connections.

I know deadline is coming closer, customer is in a bad mood and the
requirements are doubled during the project, but I recommend to avoid
any SQL and Busines Logic in JSP. They are hard to debug and somehow
that's dirty. IMHO JSP it's for presentation.

my 0.02 cents...

  ~Gerhard

"things, that try to look like things, do often more look "

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Glenn Nielsen]

Glenn Nielsen wrote:
> 
> "Waldhoff, Rodney" wrote:
> >
> > As you probably already know, the NoSuchElementException is telling you the
> > pool has been exhausted (and the "detail" message suggests you've got the
> > pool configured to block for some finite amount of time before giving up).
> > The most probable causes, in my experience:
> >
> > 1. A peak in connection usage, causing the pool to be exhausted "naturally",
> > (e.g,. you've capped the pool at 10 instances, and you suddenly need to use
> > 11 at once).
> >
> > 2. A slow leak, like Michael described:
> >
> > > I suspect that you may have a pool leak.  Somewhere
> > > you're borrowing from the pool and not returning it
> > > to the pool.  Possibly the return of an object is
> > > not in a finally block and an rare exception causes
> > > it to lose one object.
> >
> > If you can't tie the exception to activity in one way or another, then it's
> > probably the latter.  Configurating the pool to "when exahausted, grow"
> > would make the problem go away (at least as long as the database can support
> > the traffic).  Setting the pool cap very low will make it happen more
> > frequently, which may make it easier to diagnose and/or determine if the
> > problem has been fixed.
> >
> > Like Michael suggested, look for places where the connection.close() method
> > isn't happenning in a finally block, and that the close call will always
> > execute within that block.  (E.g., I sometimes see code like:
> >
> > } finally {
> >   resultset.close();
> >   statement.close();
> >   connection.close();
> > }
> >
> > which won't do what we want if either of the first two lines throws an
> > Exception.  I use this idiom:
> >
> > } finally {
> >   try { resultset.close(); } catch(Exception e) { }
> >   try { statement.close(); } catch(Exception e) { }
> >   try { connection.close(); } catch(Exception e) { }
> > }
> >
> > and don't worry about null or anything else.)
> >
> 
> I have reviewed the customers JSP pages and found one page
> where they were not wrapping their use of a db connection within a
> try {} block.  I have informed them how to fix it.
> 
> But this one page does not explain running out of connections in a pool
> with a max of 75 connections.  In the logs that page only failed twice
> during the run of Tomcat before it ran out of db connections.
> 

The remainder of the pages use the DbTags JSP tag library.  I found
one JSP page in use which was not closing the dbTags closeConnection tag
to close the db connection.  This page was not frequently used, but 
probably explains the slow connection leak.

Thanks for everyones help.

Glenn

--
Glenn Nielsen [EMAIL PROTECTED] | /* Spelin donut madder|
MOREnet System Programming   |  * if iz ina coment.  |
Missouri Research and Education Network  |  */   |
--

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Juozas Baliuka]

>
> I have reviewed the customers JSP pages and found one page
> where they were not wrapping their use of a db connection within a
> try {} block.  I have informed them how to fix it.
>
> But this one page does not explain running out of connections in a pool
> with a max of 75 connections.  In the logs that page only failed twice
> during the run of Tomcat before it ran out of db connections.
>
> Thanks,
>
> Glenn
If you find a single fragmet of code this kind, this says a lot of abaut
applications quality.
And As I understand this application uses jdbc code in jsp page.
Trust me, I made a lot of bugs in my applications and I can recomend only
 refactoring for this application.


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Glenn Nielsen]

"Waldhoff, Rodney" wrote:
> 
> As you probably already know, the NoSuchElementException is telling you the
> pool has been exhausted (and the "detail" message suggests you've got the
> pool configured to block for some finite amount of time before giving up).
> The most probable causes, in my experience:
> 
> 1. A peak in connection usage, causing the pool to be exhausted "naturally",
> (e.g,. you've capped the pool at 10 instances, and you suddenly need to use
> 11 at once).
> 
> 2. A slow leak, like Michael described:
> 
> > I suspect that you may have a pool leak.  Somewhere
> > you're borrowing from the pool and not returning it
> > to the pool.  Possibly the return of an object is
> > not in a finally block and an rare exception causes
> > it to lose one object.
> 
> If you can't tie the exception to activity in one way or another, then it's
> probably the latter.  Configurating the pool to "when exahausted, grow"
> would make the problem go away (at least as long as the database can support
> the traffic).  Setting the pool cap very low will make it happen more
> frequently, which may make it easier to diagnose and/or determine if the
> problem has been fixed.
> 
> Like Michael suggested, look for places where the connection.close() method
> isn't happenning in a finally block, and that the close call will always
> execute within that block.  (E.g., I sometimes see code like:
> 
> } finally {
>   resultset.close();
>   statement.close();
>   connection.close();
> }
> 
> which won't do what we want if either of the first two lines throws an
> Exception.  I use this idiom:
> 
> } finally {
>   try { resultset.close(); } catch(Exception e) { }
>   try { statement.close(); } catch(Exception e) { }
>   try { connection.close(); } catch(Exception e) { }
> }
> 
> and don't worry about null or anything else.)
> 

I have reviewed the customers JSP pages and found one page
where they were not wrapping their use of a db connection within a
try {} block.  I have informed them how to fix it.

But this one page does not explain running out of connections in a pool
with a max of 75 connections.  In the logs that page only failed twice
during the run of Tomcat before it ran out of db connections.

Thanks,

Glenn

--
Glenn Nielsen [EMAIL PROTECTED] | /* Spelin donut madder|
MOREnet System Programming   |  * if iz ina coment.  |
Missouri Research and Education Network  |  */   |
--

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Juozas Baliuka]

> Juozas wrote:
>
> > It is very dificult to find pooling
> > related bugs  in application. May be
> > it has meaning to implement something
> > like "DebugPool" to help find problems
> > in users code. I can think about this,
> > if this idea is interesting.
>
> In the pool/dbcp implementation from which commons-pool/commons-dbcp is
> derived, probably the most useful diagnostic we found, especially for the
> kinds of problems Glenn seems to be having, is to track the number of
> connections being used by each thread (using a ThreadLocal variable), and
> logging or throwing an exception when that number is greater than 1.  In
all
> cases I can think of when a thread is borrowing more than one connection
> from the same pool (or from mutliple pools) it is either a logic or design
> error.
>
> We ripped out that, and some logging, out of concern for the general case,
> but I like the idea of a DebugPool.  Perhaps DebugPool could simply be a
> thin wrapper around an arbitrary pool, adding logging and related
debugging
> calls.
This idea is very dirty at this time:
1) "Remember" threat and stack trace then Thread obtains pooled object
2) Log stack trace if Thread "dead", but not returned object to pool.

It will not help for all situations, because the stack tracecan be always
the same for some applications,
But this kind of applications never have problems with pool :)
It is always possible to implement jdbc code in this way:

execute(  SQL s, params[] p  ,  Callback c  ){
 /*   open   */

/* execute */

/* finally   close  */
}

and use this method for all queries and updates








--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Juozas Baliuka]

 
> which won't do what we want if either of the first two lines throws an
> Exception.  I use this idiom:
> 
> } finally {
>   try { resultset.close(); } catch(Exception e) { }
>   try { statement.close(); } catch(Exception e) { }
>   try { connection.close(); } catch(Exception e) { }
> }
> 
> and don't worry about null or anything else.)
> 
>  - Rod
Yes it is good idea. I can recoment something like this

void closeJDBCObject(Object obj){
try{
 if (obj != null){ // don't need this
if ( obj instanceof Connection ){
 ((Connection)obj).close();
  }
   else if ( obj instanceof ResultSet ){
 ...

  }
}catch( Throwable  ignore){

 }

} 


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

RE: commons dbcp or pool problems

[Waldhoff, Rodney]

Juozas wrote:

> It is very dificult to find pooling 
> related bugs  in application. May be 
> it has meaning to implement something 
> like "DebugPool" to help find problems
> in users code. I can think about this, 
> if this idea is interesting.

In the pool/dbcp implementation from which commons-pool/commons-dbcp is
derived, probably the most useful diagnostic we found, especially for the
kinds of problems Glenn seems to be having, is to track the number of
connections being used by each thread (using a ThreadLocal variable), and
logging or throwing an exception when that number is greater than 1.  In all
cases I can think of when a thread is borrowing more than one connection
from the same pool (or from mutliple pools) it is either a logic or design
error.

We ripped out that, and some logging, out of concern for the general case,
but I like the idea of a DebugPool.  Perhaps DebugPool could simply be a
thin wrapper around an arbitrary pool, adding logging and related debugging
calls.

Re: commons dbcp or pool problems

[Juozas Baliuka]

> Then again, I haven't looked at the pool component, so I don't know
> whether this would be tye symptom for that problem or not.  Take what I
> say with a grain of salt.  Although it might be a good place to start
> looking.
>
> regards,
> michael
>

Hi,
It is very dificult to find pooling related bugs  in application.
May be it has meaning to implement something like "DebugPool" to help find
problems
in users code.
I can think about this, if this idea is interesting.


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

RE: commons dbcp or pool problems

[Waldhoff, Rodney]

As you probably already know, the NoSuchElementException is telling you the
pool has been exhausted (and the "detail" message suggests you've got the
pool configured to block for some finite amount of time before giving up).
The most probable causes, in my experience:

1. A peak in connection usage, causing the pool to be exhausted "naturally",
(e.g,. you've capped the pool at 10 instances, and you suddenly need to use
11 at once).

2. A slow leak, like Michael described:

> I suspect that you may have a pool leak.  Somewhere 
> you're borrowing from the pool and not returning it 
> to the pool.  Possibly the return of an object is 
> not in a finally block and an rare exception causes 
> it to lose one object.

If you can't tie the exception to activity in one way or another, then it's
probably the latter.  Configurating the pool to "when exahausted, grow"
would make the problem go away (at least as long as the database can support
the traffic).  Setting the pool cap very low will make it happen more
frequently, which may make it easier to diagnose and/or determine if the
problem has been fixed.  

Like Michael suggested, look for places where the connection.close() method
isn't happenning in a finally block, and that the close call will always
execute within that block.  (E.g., I sometimes see code like:

} finally {
  resultset.close();
  statement.close();
  connection.close();
}

which won't do what we want if either of the first two lines throws an
Exception.  I use this idiom:

} finally {
  try { resultset.close(); } catch(Exception e) { }
  try { statement.close(); } catch(Exception e) { }
  try { connection.close(); } catch(Exception e) { }
}

and don't worry about null or anything else.)

 - Rod

Re: commons dbcp or pool problems

[Glenn Nielsen]

"Michael A. Smith" wrote:
> 
> On Fri, 22 Feb 2002, Glenn Nielsen wrote:
> > I have been using Tomcat 4.1-dev and its new DBCP DataSource Factory
> > which uses commons-dbcp, which then uses other commons components
> > like commons-pool.
> >
> > Everything works fine, then all of a sudden GenericPool can no longer
> > get a db connection from the pool.  All attempts to get a connection
> > from the pool fail.  The only way to fix the problem is to stop and
> > restart Tomcat.  In case you ask, the db is running fine, and an
> > application which uses Torque to manage db connections continues
> > to work.  Only the commons-dbcp exhibits the failure.
> >
> > This has happen twice in the last 24 hours, both times were about 10 - 12
> > hours after Tomcat started up.  The first run was during the day with higher
> > traffic, the second was overnight with lower traffic.  During each run the
> > DBCP has handled many thousands of connection requests.
> >
> > Here is a snippet from the exception:
> >
> > 2002-02-22 08:47:53 StandardWrapperValve[jsp]: Servlet.service() for servlet jsp 
>threw
> > exception
> > java.util.NoSuchElementException: Timeout waiting for idle object
> > at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Unknown 
>Source)
> > at org.apache.commons.dbcp.PoolingDataSource.getConnection(Unknown Source)
> > at
> > org.apache.taglibs.dbtags.connection.ConnectionTag.doEndTag(ConnectionTag.java:214)
> > at org.apache.jsp.pastnumsearch$jsp._jspService(pastnumsearch$jsp.java:116)
> >
> 
> Without knowing any details about the pool component (yet -- I plan on
> giving it a good look eventually), I suspect that you may have a pool
> leak.  Somewhere you're borrowing from the pool and not returning it to
> the pool.  Possibly the return of an object is not in a finally block and
> an rare exception causes it to lose one object.
> 
> Then again, I haven't looked at the pool component, so I don't know
> whether this would be tye symptom for that problem or not.  Take what I
> say with a grain of salt.  Although it might be a good place to start
> looking.
> 

Yes, that is something I looked at.  This dbcp is only used by dbtags
in JSP pages.  During these runs a JSP page failed only a couple of times
before the general dbcp connection pool failure started.  So this isn't
due to a JSP page failure causing a connection leak.

I am also looking into the JDBC driver and the DbTags JSP tag library.

The JSP Page Author is also going to be reviewing the code for his usage
of DbTags in his JSP pages.

Regards,

Glenn

--
Glenn Nielsen [EMAIL PROTECTED] | /* Spelin donut madder|
MOREnet System Programming   |  * if iz ina coment.  |
Missouri Research and Education Network  |  */   |
--

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: commons dbcp or pool problems

[Michael A. Smith]

On Fri, 22 Feb 2002, Glenn Nielsen wrote:
> I have been using Tomcat 4.1-dev and its new DBCP DataSource Factory
> which uses commons-dbcp, which then uses other commons components
> like commons-pool.
> 
> Everything works fine, then all of a sudden GenericPool can no longer
> get a db connection from the pool.  All attempts to get a connection
> from the pool fail.  The only way to fix the problem is to stop and
> restart Tomcat.  In case you ask, the db is running fine, and an
> application which uses Torque to manage db connections continues
> to work.  Only the commons-dbcp exhibits the failure.
> 
> This has happen twice in the last 24 hours, both times were about 10 - 12
> hours after Tomcat started up.  The first run was during the day with higher
> traffic, the second was overnight with lower traffic.  During each run the
> DBCP has handled many thousands of connection requests.
>
> Here is a snippet from the exception:
> 
> 2002-02-22 08:47:53 StandardWrapperValve[jsp]: Servlet.service() for servlet jsp 
>threw
> exception
> java.util.NoSuchElementException: Timeout waiting for idle object
> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Unknown 
>Source)
> at org.apache.commons.dbcp.PoolingDataSource.getConnection(Unknown Source)
> at
> org.apache.taglibs.dbtags.connection.ConnectionTag.doEndTag(ConnectionTag.java:214)
> at org.apache.jsp.pastnumsearch$jsp._jspService(pastnumsearch$jsp.java:116)
> 

Without knowing any details about the pool component (yet -- I plan on
giving it a good look eventually), I suspect that you may have a pool
leak.  Somewhere you're borrowing from the pool and not returning it to
the pool.  Possibly the return of an object is not in a finally block and
an rare exception causes it to lose one object.

Then again, I haven't looked at the pool component, so I don't know
whether this would be tye symptom for that problem or not.  Take what I 
say with a grain of salt.  Although it might be a good place to start 
looking.

regards,
michael


> I am using the following commons components.
> 
> commons-collections V1.0 release from July 14, 2001.
> commons-pool nightly from Feb 1, 2002.
> commons-dbcp nightly from Feb 1, 2002.
> 
> Does anyone know of any bugs or changes in any of these which may
> be causing the general failure of pooling the dbcp connections?
> 
> Thanks,
> 
> Glenn
> 
> --
> Glenn Nielsen [EMAIL PROTECTED] | /* Spelin donut madder|
> MOREnet System Programming   |  * if iz ina coment.  |
> Missouri Research and Education Network  |  */   |
> --
> 
> --
> To unsubscribe, e-mail:   
> For additional commands, e-mail: 
> 


--
To unsubscribe, e-mail:   
For additional commands, e-mail: