Re: Tomcat jdbc connections

2022-01-26 Thread Phil Steitz




On 1/25/22 1:44 AM, Alan F wrote:

Hi Chris,

Thankyou so much for your time and detail here. I had been working on this 
yesterday and posted my findings below. In the end It turned out to be my lack 
of understanding on Tomcat, but hey we are always learning!


I would just like to update on my discovery of the issue I had with Tomcat 
resetting connections to DB. Main issue being lack of familiarity of parameters 
and how connection pooling worked.

One thing I was unable to see without using a DBA admin was the connections 
resetting, no level of logging in Tomcat seemed to reveal this, and in the end 
I used a simple netstat query to check connections.

watch -n1 "netstat -ant | grep ':1521.*ESTABLISHED' | nl | tail -n20  (last 
number depends on how many connections to monitor.)

Obvs local ports will change outgoing therefore highlighting a new connection 
when these change.

Once I had this I was able to tinker with current settings until I discovered 
the issue. Which in reality wasn't an issue per se its just that I discovered a 
non live Tomcat was behaving differently with pooled connections over a live 
host!

Just to Recap

Symptom:
Tomcat closing DBCP pooled connections  to Database every 60 seconds

Cause:
Tomcat was idle, therefore the configuration param was causing idle connections 
to be reset the period specified below (60 secs)

timeBetweenEvictionRunsMillis=6

Resolution:
The behaviour above is by design, a busy server will be less likely to trigger 
this due to connections being used!
Are you sure you did not set minEvictableIdleTimeMillis somewhere? That 
is the setting that determines how long a connection can sit idle in the 
pool before it is eligible for eviction.


Phil




-Original Message-
From: Christopher Schultz 
Sent: 24 January 2022 22:42
To: users@tomcat.apache.org
Subject: Re: Tomcat jdbc connections

Alan,

On 1/23/22 09:17, Alan F wrote:

Can I just follow up here what would be the next steps how would I go
about capturing the root cause of these very short connection times to
Oracle from Tomcat.

Honestly, I would want to know what query or queries are being run by these 
short-lived connections. Something tells me that if you are able to audit those 
queries, you'll immediately know what's going on.

SELECT * FROM query_details WHERE connection_id='deadbeef'

user   | start_time | end_time | query
zabbix | 12:00:00   | 12:00:00 | SELECT COUNT(*) FROM some_queue
zabbix | 12:01:00   | 12:01:00 | SELECT COUNT(*) FROM some_queue
zabbix | 12:02:00   | 12:02:00 | SELECT COUNT(*) FROM some_queue
zabbix | 12:03:00   | 12:03:00 | SELECT COUNT(*) FROM some_queue
zabbix | 12:04:00   | 12:04:00 | SELECT COUNT(*) FROM some_queue
zabbix | 12:05:00   | 12:05:00 | SELECT COUNT(*) FROM some_queue

/me says "Oh, right. We have monitoring."


Would it be along the lines of Wireshark or TCP dump to see what's
occurring as I gather this won't be captured in tomcat logging via
Catalina.out? Or can it be.

I would only resort to reading TCP dumps if all else fails. Why?

1. TCP dumps are large and "expensive"
2. You are encrypting connections to your database... right?!

Knowing the nature of the queries will help. If you see no queries being executed, then 
no TCP dump will help you because you won't be able to prove what component is making 
those connection anyway (unless dbcp2 pushes some environmental information over to 
Oracle whenever it makes a connection like "I'm connecting to Oracle on behalf of 
application X on Tomcat, but I'm not going to execute any queries mmm'kay"). I don't 
think dbcp does that kind of thing, so you'd have to crank-up the logging level on that 
application and/or Tomcat instance to see what's happening with the connection pool.

Hope that helps,
-chris


-----Original Message-
From: Phil Steitz 
Sent: 21 January 2022 17:50
To: users@tomcat.apache.org
Subject: Re: Tomcat jdbc connections



On 1/21/22 9:28 AM, Alan F wrote:

Ok thanks Phil ok I checked other connections in the same host and see minIdle="2" and 
initialSize="7"

Ive run a diff on this server.xml between our active prod hosts which shows 
connections on Toad up for at least a day as to this idle server reconnecting 
after minutes!  And diff is identical apart from Cluster ips.

Most likely culprit is network or other issue causing validations to fail.  
With testWhileIdle on, idle connections in the pool will be tested when visited 
and closed if validation fails.

Phil




-Original Message-
From: Phil Steitz 
Sent: 21 January 2022 16:10
To: users@tomcat.apache.org
Subject: Re: Tomcat jdbc connections



On 1/21/22 8:19 AM, Alan F wrote:

Thanks John,

Here is an example of a connection below I see
timeBetweenEvictionRunsMillis but not maxConnLifetimeMillis if the
server has no traffic does this mean




So above does that mean every 60 secs Eviction runs, does this mean a server 
with no 

Re: Tomcat jdbc connections

2022-01-21 Thread Phil Steitz




On 1/21/22 9:28 AM, Alan F wrote:

Ok thanks Phil ok I checked other connections in the same host and see minIdle="2" and 
initialSize="7"

Ive run a diff on this server.xml between our active prod hosts which shows 
connections on Toad up for at least a day as to this idle server reconnecting 
after minutes!  And diff is identical apart from Cluster ips.
Most likely culprit is network or other issue causing validations to 
fail.  With testWhileIdle on, idle connections in the pool will be 
tested when visited and closed if validation fails.


Phil





-Original Message-----
From: Phil Steitz 
Sent: 21 January 2022 16:10
To: users@tomcat.apache.org
Subject: Re: Tomcat jdbc connections



On 1/21/22 8:19 AM, Alan F wrote:

Thanks John,

Here is an example of a connection below I see
timeBetweenEvictionRunsMillis but not maxConnLifetimeMillis if the
server has no traffic does this mean




So above does that mean every 60 secs Eviction runs, does this mean a server 
with no traffic evicts unused connections to DB?

One more note on this config.  Since you have not specified minIIdle or 
initialSize, the pool will not create connections until they are requested by 
clients.

Phil


-Original Message-
From: john.e.gr...@wellsfargo.com.INVALID

Sent: 21 January 2022 14:50
To: users@tomcat.apache.org
Subject: RE: Tomcat jdbc connections

Alan,



-Original Message-
From: Alan F 
Sent: Friday, January 21, 2022 6:53 AM
To: Tomcat Users List 
Subject: RE: Tomcat jdbc connections

Hi Christopher

Thanks for your time here.

You mean like, a connection is made, no queries are executed, and
then the connection is terminated?
- ANSWER -  DBAs are saying connections last a minute or so and are
replaced by a new set of connections 1 session each pool.


Presumably, you have an application running on Tomcat with a JDBC
connection configured. Are you using Tomcat's built-in pooling, or is
your application managing its own pooling/connections?
- ANSWER we are using dbcp2


Do you have any background tasks (in the JVM) that will run even when
there is no user activity? Cache-management? Lazy-writes?
- ANSWER I don't think so.


Are you using Tomcat's clustering? Are you using a database-backed
session management or any kind? -
- ANSWER YES clustered B node down, A node up this Tomcat node is not
live or receiving any traffic currently in test.


Are the connections definitely being made from the application itself
and/or Tomcat? Or do you just see those connections coming from the
IP where the application is running? - ANSWER im assuming according
to server.xml they are utlising these declared connections via Tomcat.


Do you have any background tasks (outside the JVM) that will run even
when there are no user actions? For example, some monitoring system
that pings the server to say "can you reach the database?" - ANSWER
NO


O


Could your pool be closing the connections after a set time?  Look at 
timeBetweenEvictionRunsMillis and maxConnLifetimeMillis here:

https://commons.apache.org/proper/commons-dbcp/configuration.html
B CB  [  
X  ܚX KK[XZ[
   \ \  ][  X  ܚX P X ]
   \X K ܙ B  ܈Y][ۘ[  [X[  K[XZ[
   \ \  Z[ X ]
   \X K ܙ B

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat jdbc connections

2022-01-21 Thread Phil Steitz




On 1/21/22 8:19 AM, Alan F wrote:

Thanks John,

Here is an example of a connection below I see timeBetweenEvictionRunsMillis 
but not maxConnLifetimeMillis if the server has no traffic does this mean




So above does that mean every 60 secs Eviction runs, does this mean a server 
with no traffic evicts unused connections to DB?


One more note on this config.  Since you have not specified minIIdle or 
initialSize, the pool will not create connections until they are 
requested by clients.


Phil



-Original Message-
From: john.e.gr...@wellsfargo.com.INVALID 
Sent: 21 January 2022 14:50
To: users@tomcat.apache.org
Subject: RE: Tomcat jdbc connections

Alan,



-Original Message-
From: Alan F 
Sent: Friday, January 21, 2022 6:53 AM
To: Tomcat Users List 
Subject: RE: Tomcat jdbc connections

Hi Christopher

Thanks for your time here.

You mean like, a connection is made, no queries are executed, and then
the connection is terminated?
- ANSWER -  DBAs are saying connections last a minute or so and are
replaced by a new set of connections 1 session each pool.


Presumably, you have an application running on Tomcat with a JDBC
connection configured. Are you using Tomcat's built-in pooling, or is
your application managing its own pooling/connections?
- ANSWER we are using dbcp2


Do you have any background tasks (in the JVM) that will run even when
there is no user activity? Cache-management? Lazy-writes?
- ANSWER I don't think so.


Are you using Tomcat's clustering? Are you using a database-backed
session management or any kind? -
- ANSWER YES clustered B node down, A node up this Tomcat node is not
live or receiving any traffic currently in test.


Are the connections definitely being made from the application itself
and/or Tomcat? Or do you just see those connections coming from the IP
where the application is running? - ANSWER im assuming according to
server.xml they are utlising these declared connections via Tomcat.


Do you have any background tasks (outside the JVM) that will run even
when there are no user actions? For example, some monitoring system
that pings the server to say "can you reach the database?" - ANSWER NO


O


Could your pool be closing the connections after a set time?  Look at 
timeBetweenEvictionRunsMillis and maxConnLifetimeMillis here:

https://commons.apache.org/proper/commons-dbcp/configuration.html
B CB  [  
X  ܚX KK[XZ[
  \ \  ][  X  ܚX P X ]
  \X K ܙ B  ܈Y][ۘ[  [X[  K[XZ[
  \ \  Z[ X ]
  \X K ܙ B

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat jdbc connections

2022-01-21 Thread Phil Steitz



On 1/21/22 8:19 AM, Alan F wrote:

Thanks John,

Here is an example of a connection below I see timeBetweenEvictionRunsMillis 
but not maxConnLifetimeMillis if the server has no traffic does this mean




So above does that mean every 60 secs Eviction runs, does this mean a server 
with no traffic evicts unused connections to DB?
Not until they have been idle for minEvictableIdleTimeMillis ms. Since 
you have not specified this property, that will default to 30 mins (see 
[1]), so while the check will run every 60 secs, no connections should 
be removed for being idle too long until they have been idle in the pool 
for 30 mins.   One thing that could cause connections to get closed by 
the pool under the config above is validation failures.  Can your DBAs 
see the validation queries succeeding?


Phil

[1] https://commons.apache.org/proper/commons-dbcp/configuration.html


-Original Message-
From:john.e.gr...@wellsfargo.com.INVALID
Sent: 21 January 2022 14:50

To:users@tomcat.apache.org
Subject: RE: Tomcat jdbc connections

Alan,



-Original Message-
From: Alan F
Sent: Friday, January 21, 2022 6:53 AM
To: Tomcat Users List
Subject: RE: Tomcat jdbc connections

Hi Christopher

Thanks for your time here.

You mean like, a connection is made, no queries are executed, and then
the connection is terminated?
- ANSWER -  DBAs are saying connections last a minute or so and are
replaced by a new set of connections 1 session each pool.


Presumably, you have an application running on Tomcat with a JDBC
connection configured. Are you using Tomcat's built-in pooling, or is
your application managing its own pooling/connections?
- ANSWER we are using dbcp2


Do you have any background tasks (in the JVM) that will run even when
there is no user activity? Cache-management? Lazy-writes?
- ANSWER I don't think so.


Are you using Tomcat's clustering? Are you using a database-backed
session management or any kind? -
- ANSWER YES clustered B node down, A node up this Tomcat node is not
live or receiving any traffic currently in test.


Are the connections definitely being made from the application itself
and/or Tomcat? Or do you just see those connections coming from the IP
where the application is running? - ANSWER im assuming according to
server.xml they are utlising these declared connections via Tomcat.


Do you have any background tasks (outside the JVM) that will run even
when there are no user actions? For example, some monitoring system
that pings the server to say "can you reach the database?" - ANSWER NO


O


Could your pool be closing the connections after a set time?  Look at 
timeBetweenEvictionRunsMillis and maxConnLifetimeMillis here:

https://commons.apache.org/proper/commons-dbcp/configuration.html
B CB  [  
X  ܚX KK[XZ[
  \ \  ][  X  ܚX P X ]
  \X K ܙ B  ܈Y][ۘ[  [X[  K[XZ[
  \ \  Z[ X ]
  \X K ܙ B

-
To unsubscribe, e-mail:users-unsubscr...@tomcat.apache.org
For additional commands, e-mail:users-h...@tomcat.apache.org



Re: RemoveAbandoned Problems

2021-12-08 Thread Phil Steitz




On 12/8/21 1:44 PM, Christopher Schultz wrote:

Phil,

On 12/8/21 15:23, Phil Steitz wrote:



On 12/8/21 6:36 AM, Christopher Schultz wrote:

Jerry,

On 12/7/21 20:59, Jerry Malcolm wrote:
Chris, The way I thought it worked was if I configured 
'RemoveAbandonedOnBorrow' and RemoveAbandonedTimeout="15" was that 
each time I requested a new connection from the pool, any 
connections that had been idle for >15 minutes and had not been 
returned by my code to the pool would be recovered, returned to the 
pool and logged (assuming logAbandoned was set).


Nope. "removeAbandoned" causes any connection that isn't returned to 
the pool to be *removed from the pool*, and replaced with a new 
(presumably working) connection. The connection that was never 
returned ... stays out there, doing whatever it was doing.


Not exactly.  Abandoned connection cleanup does try to physically 
close connections [1] that are deemed abandoned.  It creates capacity 
to create new ones but it does not create them immediately.


"logAbandoned" just lets you know when the pool gives up. It doesn't 
"do" anything (other than the logging).


The alternative would be for the pool to forcibly terminate the 
connection, which could cause all kinds of chaos, so it does the 
only thing it can reasonably do: forget the connection ever existed 
in the first place. If your code never closes it, and the Connection 
object never gets GC'd (and, presumably, closed in the process), 
then it just lived forever, wasting an open-connection to your db. 
Since you have limited your total connections (per user? per host?) 
you eventually run out due to the leak.


This is why you need to be careful to set the abandoned timeout long 
enough so that "chaos" does not ensue.  The pool tries to physically 
close abandoned connections when this is configured to happen.  If 
clients retain handles to them and later try to use them, they will 
get exceptions.  You can see this confirmed in DBCP's 
TestAbandonedBasicDataSource unit tests.


Oh, I had no idea DBCP was actually trying to kill those connections. 
I guess reasonable can disagree over whether or not those connections 
should be killed by DBCP.


Yeah it has been that way since inception.  The problem with not 
actually closing them is you end up creating problems on the DB side if 
you leave them hanging.  Also kind of defeats the purpose of the pool 
and changes its contract vis a vis the factory (maxTotal means a pool 
will not take up more than that many instances at a given time).


Phil


I'm cool either way :)

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: RemoveAbandoned Problems

2021-12-08 Thread Phil Steitz




On 12/8/21 6:36 AM, Christopher Schultz wrote:

Jerry,

On 12/7/21 20:59, Jerry Malcolm wrote:
Chris, The way I thought it worked was if I configured 
'RemoveAbandonedOnBorrow' and RemoveAbandonedTimeout="15" was that 
each time I requested a new connection from the pool, any connections 
that had been idle for >15 minutes and had not been returned by my 
code to the pool would be recovered, returned to the pool and logged 
(assuming logAbandoned was set).


Nope. "removeAbandoned" causes any connection that isn't returned to 
the pool to be *removed from the pool*, and replaced with a new 
(presumably working) connection. The connection that was never 
returned ... stays out there, doing whatever it was doing.


Not exactly.  Abandoned connection cleanup does try to physically close 
connections [1] that are deemed abandoned.  It creates capacity to 
create new ones but it does not create them immediately.


"logAbandoned" just lets you know when the pool gives up. It doesn't 
"do" anything (other than the logging).


The alternative would be for the pool to forcibly terminate the 
connection, which could cause all kinds of chaos, so it does the only 
thing it can reasonably do: forget the connection ever existed in the 
first place. If your code never closes it, and the Connection object 
never gets GC'd (and, presumably, closed in the process), then it just 
lived forever, wasting an open-connection to your db. Since you have 
limited your total connections (per user? per host?) you eventually 
run out due to the leak.


This is why you need to be careful to set the abandoned timeout long 
enough so that "chaos" does not ensue.  The pool tries to physically 
close abandoned connections when this is configured to happen.  If 
clients retain handles to them and later try to use them, they will get 
exceptions.  You can see this confirmed in DBCP's 
TestAbandonedBasicDataSource unit tests.


Until a few days ago I had a code error that was bypassing the 
closing of the connection in certain situations, and after 12-24 
hours the pool had worked its way up to maxing out.  My problem is 
fixed now, and the numActive count is staying fairly flat during 
normal activity.  But the way I understood removeAbandonedOnBorrow 
was that TC connection pooling code would not allow errant 
connections to remain in use forever.


I'm sure I'm just misunderstanding how it works.  Again, not critical 
at this moment.  But I'd like to figure out where my understanding is 
wrong for future situations.


I think that the reason that you did not see connections closed by the 
pool may have been that you did not get close enough to the maxTotal 
setting (see other response above) or you had idle connections in the 
pool as it was leaking and you did not have 
timeBetweenEvictionRunsMillis set.  If you set 
timeBetweenEvictionRunsMillis to a positive value, that will trigger 
unconditional removal when it runs (i.e., it does not check how close 
the pool is to maxTotal or how many idle connections there are).  The 
removeAbandonedOnBorrow setting is really more of a liveness than a 
cleanup feature - basically trying to keep the pool ahead of demand by 
cleaning up when it is exhausted or close to it.


Phil

[1] As of DBCP 2.9.0, abort is used in place of close.  See 
https://issues.apache.org/jira/browse/DBCP-567




You thought the pool would "clean-up" the mess. IT doesn't. What it 
*does* do is allow the pool to continue to function and provide its 
service to the application, even when the application is leaking 
connections.


So, rather than starving clients when connections leak, those 
connections are simply allowed to leak.


I always recommend running with maxActive="1" in development, with 
removeAbandoned="false" and logAbandoned="true". You'll find any leaks 
VERY quickly. ;)


-chris


On 12/7/2021 2:31 PM, Christopher Schultz wrote:

Jerry,

On 12/4/21 23:06, Jerry Malcolm wrote:
I had a db connection leak in my code where an error condition 
would throw an exception and bypass the connection cleanup code. I 
found that and fixed it.  But before I found the problem, my 
program was overrunning the max connections and locking out.  It 
would take sometimes 12 hours after a reboot to go from 0 
connections to max. Normal steady state connections should 
currently be under 50.  The ramp over several hours to max was very 
obvious in my numActive log. What I'm confused about is why 
removeAbandoned didn't recover those connections.


When you say "recover"... what exactly do you mean?

Granted, if I write my code correctly, removeAbandoned shouldn't be 
necessary. The coding problem is solved now.  But apparently my 
understanding/configuration of removeAbandoned is not correct.


Possibly, but you didn't state your expectations.

I'd like to have that figured out in case there's a next time 
(which sadly there probably will be). Basically, with the 
configuration below, I'm not getting any idle connections 

Re: RemoveAbandoned Problems

2021-12-06 Thread Phil Steitz

Sorry again.  Docs are here (at the bottom in the abandoned config section):

https://commons.apache.org/proper/commons-dbcp/configuration.html

On 12/6/21 10:01 AM, Phil Steitz wrote:



On 12/5/21 2:34 PM, Jerry Malcolm wrote:

Phil,

Thanks for the response.  I saw that note in the docs that said the 
removeAbandonedOnMaintenance wouldn't do anything without an evictor 
service.  But removeAbandonedOnBorrow also requires an evictor 
service to run in order remove on borrow?  That's fine. Just a bit 
confusing that on-borrow requires a timed eviction run.  I'll do 
whatever it takes.  Again, just trying to figure it all out.


I am sorry.  My mistake.  I saw the removeAbandonOnMaintenance and not 
the other one.  You are correct that with removeAbandonedOnBorrow on 
you should not need to have the evictor turned on (though it will 
obviously only run on borrow).  The docs could be clearer on this 
(seems some have been moved / deleted), but when 
removeAbandonedOnMaintenance is on, actual removal only happens when 
there are fewer than 2 idle objects available in the pool and 
getNumActive() > getMaxTotal() - 3.


Phil


Jerry

On 12/5/2021 12:19 PM, Phil Steitz wrote:
In order for abandoned connection cleanup to happen, you have to 
have configured a maintenance (aka "evictor") thread to run.  You 
need to set the value of timeBetweenEvictionRunsMillis to a positive 
number.


Phil

On 12/4/21 9:06 PM, Jerry Malcolm wrote:
I had a db connection leak in my code where an error condition 
would throw an exception and bypass the connection cleanup code. I 
found that and fixed it.  But before I found the problem, my 
program was overrunning the max connections and locking out.  It 
would take sometimes 12 hours after a reboot to go from 0 
connections to max. Normal steady state connections should 
currently be under 50.  The ramp over several hours to max was very 
obvious in my numActive log. What I'm confused about is why 
removeAbandoned didn't recover those connections.  Granted, if I 
write my code correctly, removeAbandoned shouldn't be necessary. 
The coding problem is solved now.  But apparently my 
understanding/configuration of removeAbandoned is not correct. I'd 
like to have that figured out in case there's a next time (which 
sadly there probably will be). Basically, with the 
configuration below, I'm not getting any idle connections detected 
and returned.  This is TC 8.5.73. And the leak was happening on a 
basic request/response (no threads involved).  I requested the 
connection, encountered an error, and returned without closing the 
connection. Ideas? Thx.





-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org






-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: RemoveAbandoned Problems

2021-12-06 Thread Phil Steitz




On 12/5/21 2:34 PM, Jerry Malcolm wrote:

Phil,

Thanks for the response.  I saw that note in the docs that said the 
removeAbandonedOnMaintenance wouldn't do anything without an evictor 
service.  But removeAbandonedOnBorrow also requires an evictor service 
to run in order remove on borrow?  That's fine. Just a bit confusing 
that on-borrow requires a timed eviction run.  I'll do whatever it 
takes.  Again, just trying to figure it all out.


I am sorry.  My mistake.  I saw the removeAbandonOnMaintenance and not 
the other one.  You are correct that with removeAbandonedOnBorrow on you 
should not need to have the evictor turned on (though it will obviously 
only run on borrow).  The docs could be clearer on this (seems some have 
been moved / deleted), but when removeAbandonedOnMaintenance is on, 
actual removal only happens when there are fewer than 2 idle objects 
available in the pool and getNumActive() > getMaxTotal() - 3.


Phil


Jerry

On 12/5/2021 12:19 PM, Phil Steitz wrote:
In order for abandoned connection cleanup to happen, you have to have 
configured a maintenance (aka "evictor") thread to run.  You need to 
set the value of timeBetweenEvictionRunsMillis to a positive number.


Phil

On 12/4/21 9:06 PM, Jerry Malcolm wrote:
I had a db connection leak in my code where an error condition would 
throw an exception and bypass the connection cleanup code. I found 
that and fixed it.  But before I found the problem, my program was 
overrunning the max connections and locking out.  It would take 
sometimes 12 hours after a reboot to go from 0 connections to max.  
Normal steady state connections should currently be under 50.  The 
ramp over several hours to max was very obvious in my numActive log. 
What I'm confused about is why removeAbandoned didn't recover those 
connections.  Granted, if I write my code correctly, removeAbandoned 
shouldn't be necessary. The coding problem is solved now.  But 
apparently my understanding/configuration of removeAbandoned is not 
correct. I'd like to have that figured out in case there's a next 
time (which sadly there probably will be).  Basically, with the 
configuration below, I'm not getting any idle connections detected 
and returned.  This is TC 8.5.73.  And the leak was happening on a 
basic request/response (no threads involved).  I requested the 
connection, encountered an error, and returned without closing the 
connection.  Ideas? Thx.





-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: RemoveAbandoned Problems

2021-12-05 Thread Phil Steitz
In order for abandoned connection cleanup to happen, you have to have 
configured a maintenance (aka "evictor") thread to run.  You need to set 
the value of timeBetweenEvictionRunsMillis to a positive number.


Phil

On 12/4/21 9:06 PM, Jerry Malcolm wrote:
I had a db connection leak in my code where an error condition would 
throw an exception and bypass the connection cleanup code. I found 
that and fixed it.  But before I found the problem, my program was 
overrunning the max connections and locking out.  It would take 
sometimes 12 hours after a reboot to go from 0 connections to max.  
Normal steady state connections should currently be under 50.  The 
ramp over several hours to max was very obvious in my numActive log.  
What I'm confused about is why removeAbandoned didn't recover those 
connections.  Granted, if I write my code correctly, removeAbandoned 
shouldn't be necessary. The coding problem is solved now.  But 
apparently my understanding/configuration of removeAbandoned is not 
correct. I'd like to have that figured out in case there's a next time 
(which sadly there probably will be).  Basically, with the 
configuration below, I'm not getting any idle connections detected and 
returned.  This is TC 8.5.73.  And the leak was happening on a basic 
request/response (no threads involved).  I requested the connection, 
encountered an error, and returned without closing the connection.  
Ideas? Thx.





-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Determining My Own IP address?

2021-04-01 Thread Phil Steitz




On 4/1/21 7:05 PM, Jerry Malcolm wrote:
I am running several clustered Tomcat EC2 instances in an Amazon 
autoscaling group.  These instances come and go based on load and 
failure/restart.  IP addresses are assigned dynamically.  As far I can 
figure, the only determinable difference between the instances is the 
IP address that is assigned to each EC2 when it spins up.


I'm seeing some really weird symptoms at times that seem to indicate 
that the load balancer is not always honoring session affinity.  It 
appears that another server periodically gets sent a request for a 
session that is being handled by a different EC2. The load balancer 
problem, if it exists, is definitely not a tomcat problem.  However, 
in order to figure out what is really happening, I need to be able to 
track and differentiate which server is handling each request.  Inside 
my TC webapp, if I could determine programatically some unique 
identifier for the EC2 running that http request, I could include that 
in my logs.


My question is, is there a way to determine "my own" IP address from 
within a webapp servlet?  Or is there some other way that I can log 
some string or some value that will guarantee to be unique within each 
instance of Tomcat?  I guess another option would be to somehow get 
the tomcat bootup timestamp (from within a servlet) assuming two EC2s 
won't boot at precisely the same instant.  But I don't know a way to 
access the boot timestamp.  Am I missing something more obvious?


You can get this from the getLocalAddr method of the ServletRequest 
object passed to your servlet's doPost/doGet.  See [1].


Phil

https://javaee.github.io/javaee-spec/javadocs/javax/servlet/ServletRequest.html#getLocalAddr


Thx

Jerry


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat JDBC Connection Pool question

2021-02-10 Thread Phil Steitz

See maxWait for Tomcat JDBC Pool

https://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html

Or for the default dbcp pool, maxWaitMillis

https://commons.apache.org/proper/commons-dbcp/configuration.html

Phil

On 2/10/21 3:22 PM, xcorpius wrote:

Hi Chris!

ConnectionReserveTimeoutSeconds:

The number of seconds after which a call to reserve a connection from the 
connection pool will timeout.

When set to 0, a call will never timeout.

When set to -1, a call will timeout immediately.

Admin Console field label: Connection Reserve Timeout

Units: seconds

Default: 10

Minimum: -1

Maximum: 231-1

Thanks,

Xcorpius
 Mensaje original 
On 10 feb. 2021 23:13, Christopher Schultz escribió:


Xcorpius,

On 2/10/21 07:15, xcorpius wrote:

Is there a parameter in "Tomcat JDBC Connection Pool" 9 equivalent to
the ConnectionReserveTimeoutSeconds parameter from Weblogic?

Maybe.

What does that parameter actually do?

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: troubled by "SEVERE: Cannot register null bean"

2021-01-10 Thread Phil Steitz
The problem is what Mark pointed out above.  In the type attribute you 
use the commons package name, but then specify the tomcat factory.  Try 
changing the type attribute to


type="org.apache.tomcat.dbcp.dbcp2.datasources.PerUserPoolPoolDataSource"

Phil

On 1/10/21 8:41 AM, Rob Sargent wrote:
I've abandon PerUserPoolDataSource and retreated to 
org.apache.tomcat.jdbc.pool.DataSourceand multiple resource entries in 
context.xml (probably even if I could make PerUser work).


I'm clearly missing a crucial piece.  Here's my current full context.xml

   
      factory="org.apache.tomcat.dbcp.dbcp2.datasources.PerUserPoolDataSourceFactory" 


        validationQuery="select 1"
        testWhileIdle="false"
        testOnBorrow="true"
        testOnReturn="false"
        validationInterval="3"
        timeBetweenEvictionRunsMillis="3"
        maxActive="50"
        initialSize="5"
        maxWait="1"
        removeAbandonedTimeout="60"
        minEvictableIdleTimeMillis="3"
        minIdle="1"
        maxIdle="5"
        logAbandoned="true"
        removeAbandoned="true"
        username="x"
        password="y"
      />
   

My apologies for the noise.

On 1/9/21 8:10 PM, Rob Sargent wrote:
After many loops through this, I'm pretty certain my datasource 
factory is simply never added to the context as per the SEVERE message.



Any other combinations of lookups generates a NamingException.

This verbose approach demonstrates that the key is in the map and it 
is mapped to null.

    InitialContext ic = new InitialContext();
    Context sgsContext  = (Context) ic.lookup("java:/comp/env");
    Context jctx = (Context) sgsContext.lookup("jdbc");
    dataSource = (PerUserPoolDataSource) jctx.lookup("sgsdb");


gets to the last line with a valid "jctx" but still returns null (and 
NOT a NamingException)


    // normally I would do just two lookups
    //Context sgsContext  = (Context) ic.lookup("java:/comp/env");
    //dataSource = (PerUserPoolDataSource) 
sgsContext.lookup("jdbc/sgsdb");



In NamingContextListener.addResource(ContextResource resorce) 
(9.0.41), Lines 1011-1028


   if (("javax.sql.DataSource".equals(ref.getClassName())  ||
    "javax.sql.XADataSource".equals(ref.getClassName())) &&
    resource.getSingleton()) {
    Object actualResource =null;
    try {
    ObjectName on = createObjectName(resource);
    actualResource =envCtx.lookup(resource.getName());
Registry.getRegistry(null,null).registerComponent(actualResource,on,null); 


    objectNames.put(resource.getName(),on);
    }catch (Exception e) {
log.warn(sm.getString("naming.jmxRegistrationFailed", e));
    }
    // Bug 63210. DBCP2 DataSources require an explicit close. 
This goes
   // further and cleans up and AutoCloseable DataSource by default. 
if (actualResourceinstanceof AutoCloseable && 
!resource.getCloseMethodConfigured()) {

    resource.setCloseMethod("close");
    }
   }

The lookup setting actualResource (line 1017) returns null. This uses 
the passed in resource which has name = "jdbc/sgsdb" and lookupName = 
null.  That no resource is found makes sense since it's being 
registered and registerComponent refuses to register a null value.


I have no idea why the actualResource (a DataSource Factory) is not 
available.



On 1/8/21 9:11 AM, Rob Sargent wrote:

I'm getting the following error message during startup

   SEVERE: Cannot register null bean for
[Tomcat:type=DataSource,host=localhost,context=/sgs,class=javax.sql.DataSource,name="jdbc/sgsdb"] 



with sgs/META-INF/context.xml as

   
      classname="org.apache.tomcat.dbcp.dbcp.datasources.PerUserPoolDataSource" 

factory="org.apache.commons.dbcp2.datasources.PerUserPoolDataSourceFactory" 


      />
   

and sgs/WEB-INF/web.xml as

   
   http://java.sun.com/xml/ns/j2ee;
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
     xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
   http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd;
     version="2.5">
      SGSaaS
      
        Enable debugging for the application
        debug
        true
      
      
databaseHost
${SGSSRVR_databaseHost}
java.lang.String
      
      
databasePort
${SGSSRVR_databasePort}
java.lang.Integer
      
      
roleExtension
${SGSSRVR_roleExtension}
java.lang.String
      
      
expansionStep
${SGSSRVR_expansionStep}
java.lang.Integer
      
      
        Pointer to context datasource
        jdbc/sgsdb
        javax.sql.DataSource
        Container
      
   


Deep down in NamingContextListen

   actualResource =envCtx.lookup(resource.getName());

fails, setting actualResource to null and resource.getName() is 
"jdbc/sgsdb"


I can get the env values (some of which are superfluous) in my app, 
but I cannot get the DataSource I think because it isn't there.


I've tried (with castings removed for brevity)

   ctx = initialContext.lookup("java:/comp/env");
   dataSource = 

Re: jdbc connction pool issues [EXTERNAL]

2020-12-15 Thread Phil Steitz



On 12/15/20 1:35 PM, Beard, Shawn wrote:

No intitialSize is not defined.

Im getting the data to verify from JMX, however we also have an APM called 
appdynamics loaded. Both verified the 8 max connections.

Other tomcat servers with exact same jdbc connection pool config(only 
difference is servername, databasename, user and pass) show 50 max connections, 
which is what maxActive is set to.


Are the other tomcat servers running the same tomcat version? Since TC8, 
the default connection pool is DBCP2 which uses maxTotal in place of 
maxActive.  See [1]. If you are running 8+, s/maxActive/maxTotal in the 
config should work.


Phil

[1] https://tomcat.apache.org/migration-8.html#Database_Connection_Pooling





Shawn​  Beard

Sr. Systems Engineer |
 BTS

Middleware Engineering   |  +1-515-564-2528 |  
sbe...@wrberkley.com









-Original Message-
From: Johnson, Jim 
Sent: Tuesday, December 15, 2020 2:22 PM
To: Tomcat Users List 
Subject: RE: jdbc connction pool issues [EXTERNAL]

** CAUTION: External message


Hi Shawn,

No, I don't think that maxActive means that it's defining the maximum number of 
connections for the pool, I think it's strictly referring to the "[ .. ] maximum 
number of *active connections* that can be allocated from this pool at the same time. [ 
.. ]" (emphasis mine on active connections)

Here is the doc that I've been referring to - sorry for not linking it earlier:
https://urldefense.com/v3/__https://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html__;!!Li8W9_Um1Taa!vJGrxr7yc86joO5r6QoNstpipfempGFqrYRe3_NyAIGNs-LenmBWhtjn1fRiAOex$

On those other Tomcat servers is there a initialSize defined?

Jim

-Original Message-
From: Beard, Shawn 
Sent: Tuesday, December 15, 2020 2:49 PM
To: Tomcat Users List 
Subject: RE: jdbc connction pool issues [EXTERNAL]

But wouldn’t maxActive mean the connection pool has a max of 50 conenctions? On 
other tomcat servers I have, maxActive is set to 50 and I verified im jmx that 
there are 50 max connections on the connection pool.

Here though I checked jmx, sure enough, its max connections is 8.



Shawn​ Beard

Sr. Systems Engineer |
BTS

Middleware Engineering | +1-515-564-2528 | 
sbe...@wrberkley.com









-Original Message-
From: Johnson, Jim 
Sent: Tuesday, December 15, 2020 1:35 PM
To: Tomcat Users List 
Subject: RE: jdbc connction pool issues [EXTERNAL]

** CAUTION: External message


Hi Shawn,

I think you’re missing initialSize

initialSize
(int)The initial number of connections that are created when the pool is 
started. Default value is 10

maxActive
(int) The maximum number of active connections that can be allocated from this 
pool at the same time. The default value is 100

It would make sense that 8 connections would be 80% utilized.

I would try replacing maxActive with initialSize and seeing how that works for 
you.

Good luck!

- Jim

From: Beard, Shawn 
Sent: Tuesday, December 15, 2020 2:12 PM
To: Tomcat Users List 
Subject: jdbc connction pool issues

CAUTION EXTERNAL EMAIL: This email originated from outside of the organization. 
Do not click links or open attachments unless you recognize the sender and know 
the content is safe.

We have this jdbc connection pool set up:



However we are getting this error:
Resource Pool Limit Reached
Time 12/15/20 11:48:00 AM
Summary
JDBC Connection Pool 
Catalina:class=javax.sql.DataSource,context=/XX,host=X,name="jdbc/DataSource",type=DataSource
 has reached 80% limit. Current pool size [8, Max pool size [8]

Shouldn’t the max connections of the connection pool be 50 since maxActive is 
50?



Shawn Beard • Sr. Systems Engineer
Middleware Engineering

[cid:image003.png@01D6D2E3.D5F1EFA0]

3840 109th Street Urbandale, IA 50322
Phone: +1-515-564-2528
Email: sbe...@wrberkley.com
Website: 

Re: tomcat connection pool per database (postgres)

2020-11-24 Thread Phil Steitz




On 11/24/20 8:52 AM, Rob Sargent wrote:
Perhaps I read too much into the description of "The tomcat JDBC 
Connection Pool" page?


TheJDBC Connection Pool|org.apache.tomcat.jdbc.pool|is a replacement 
or an alternative to theApache Commons DBCP 
<https://commons.apache.org/dbcp/>connection pool.


I reacted to the "replacement" bit. Are both equally sound, supported, 
surviving?


Yes.  I can't speak for jdbc-pool, but it looks like it is being 
actively maintained.  I can confirm that Commons DBCP is being 
maintained.  A repackaged, slightly stripped down version of DBCP is the 
default pool that ships with tomcat.  I am not sure if the 
PerUserPoolDataSource is included in the tomcat distro, but you can just 
use DBCP directly to get this if you want to use it.


Others who know dbcp-pool better can chime in, but I think the 
difference between the two is that DBCP has more features (including, 
for example the DS above), but those features come at the expense of a 
larger code base, dependency on Commons Pool (another widely used, 
pretty well-maintained library) and slightly worse performance.  When 
jdbc-pool was first introduced, the performance gap, and some nasty bugs 
in DBCP 1.x made the former a better choice for high-concurrency 
applications.  With DBCP 2, the gap has narrowed to the point where it 
is not practically significant.  The tomcat website text has never been 
updated to reflect this.


Phil
I try to use what I think is the safest longer term bet (my retirement 
is nigh, it shouldn't take the code with it :) )



On 11/24/20 8:28 AM, Phil Steitz wrote:


On 11/24/20 8:14 AM, Rob Sargent wrote:

Thanks. I get it. But...

- seems this solution raises the footprint of the pooler, with 
number-of-users * minimum-connection-count etc


- would it be beyond the pale for the pooler to maintain 
username-connectionList maps?


Per response elsethread, see PerUserPoolDataSource [1] provided by 
Commons DBCP.  It does that.


Phil

[1] https://s.apache.org/dlghr <https://s.apache.org/dlghr>



Thankfully, I'll be wildly successful if I have two concurrent users 
(a user may have hundreds of clients needing db connection)


(rant.  The RDBMSs really should have a more lightweight way of 
changing current user.  (e.g. postgres set role doesn't cut it, 
doesn't even invoke the users default search path))


Thanks again, I appreciate the feedback and knowledge sharing


On 11/24/20 6:28 AM, Christopher Schultz wrote:

Rob,

On 11/19/20 12:38, Rob Sargent wrote:
Since the connection URL names a specific postgres database is it 
standard practice to have a pool per target database?  (Switching 
databases in postgres amounts to closing/opening a connection.)


I generally consider a database connection pool to be a connection 
to a certain database/tablespace/schema, not a connection to an IP 
address. That's the only thing that would make sense in terms of an 
application, which would expect a connection to a specific data 
store, right?


If you are closing connections (and reopening them), the pool isn't 
dong its job.


I would recommend a separate pool per database/tablespace/schema.

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org










-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: tomcat connection pool per database (postgres)

2020-11-24 Thread Phil Steitz


On 11/24/20 8:14 AM, Rob Sargent wrote:

Thanks. I get it. But...

- seems this solution raises the footprint of the pooler, with 
number-of-users * minimum-connection-count etc


- would it be beyond the pale for the pooler to maintain 
username-connectionList maps?


Per response elsethread, see PerUserPoolDataSource [1] provided by 
Commons DBCP.  It does that.


Phil

[1] https://s.apache.org/dlghr 



Thankfully, I'll be wildly successful if I have two concurrent users 
(a user may have hundreds of clients needing db connection)


(rant.  The RDBMSs really should have a more lightweight way of 
changing current user.  (e.g. postgres set role doesn't cut it, 
doesn't even invoke the users default search path))


Thanks again, I appreciate the feedback and knowledge sharing


On 11/24/20 6:28 AM, Christopher Schultz wrote:

Rob,

On 11/19/20 12:38, Rob Sargent wrote:
Since the connection URL names a specific postgres database is it 
standard practice to have a pool per target database?  (Switching 
databases in postgres amounts to closing/opening a connection.)


I generally consider a database connection pool to be a connection to 
a certain database/tablespace/schema, not a connection to an IP 
address. That's the only thing that would make sense in terms of an 
application, which would expect a connection to a specific data 
store, right?


If you are closing connections (and reopening them), the pool isn't 
dong its job.


I would recommend a separate pool per database/tablespace/schema.

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org





Re: Details of connection pooling

2020-11-20 Thread Phil Steitz


On 11/20/20 10:25 AM, Rob Sargent wrote:
I'm using tomcat 9.0.+ and wish to use the built in connection pooling 
to connect to a postgres server.


I would like to understand the lookup mechanism for the next available 
connection. I create a context


   contextResource.setName("jdbc/sgsdb");
   contextResource.setType("javax.sql.DataSource");
   String turl =String.format("jdbc:postgresql://%s:%d/%s", host, 
port, dbname);

   contextResource.setProperty("url",turl);
contextResource.setProperty("driverClassName","org.postgresql.Driver");

which names a specific database.  Then I make a user specific connection

   DataSource ds = (DataSource)env.lookup("jdbc/sgsdb");
   conn =ds.getConnection(user, pwd);

1. To access a different database do I need to make a 
database-specific DataSource?


Yes.



2. Is the username used to locate an available connection for that 
user (maybe creating one if not found) or is the user overlain on an 
existing connection (perhaps "set role user")?


No, all connections from a (default, DBCP BasicDatacsource) pool share 
the same database credentials.


If you need separate pools per user, have a look at  the 
PerUserPoolDataSource [1] provided by Commons DBCP.


[1] https://s.apache.org/dlghr 

Phil








Re: Tomcat JDBC Pool Cleaner Deadlock Problem

2020-08-31 Thread Phil Steitz



On 8/31/20 3:21 AM, Felix Schumacher wrote:

Am 31.08.20 um 06:15 schrieb Gokhan Akgul:

Dear Phil ,
Thanks for your feedback. I forgot to mention the mysql driver version. The
Mysql driver version is 5.1.32.
My plan is to upgrade the mysql driver to 5.1.46 version and monitor for a
while.

If I read the bug report correctly, MySQL will not change its logic and
therefore using newer versions of the driver will not help.


Yeah.  There was a comment in another related bug report about changing 
the locking model in 5.1.x, so it is possible a later version will help, 
but you are right that it probably won't.




What MySQL advises is to change the pool to use the abort-Method of the
connection to close it in the case of abandoned connections.

The dbcp2 pools seems to be able to use that method, while I found no
reference to it in the jdbc-pool module (which you are using).


We are talking about making that change on commons-dev now [1], but 
currently dbcp2 uses close as jdbc-pool does.


Comments / patches welcome!

Phil



So, maybe it is a good idea to switch the used pool from the jdbc-pool
to the default tomcat pool (see
http://tomcat.apache.org/tomcat-9.0-doc/jndi-datasource-examples-howto.html).
It should work equally well (I am not sure, if it supports something
like the slowqueryreport, though). If you want to continue using the old
jdbc-pool module, you might want to file a bug on the bugtracker asking
for an enhancement to support the abort method. (I would use the dbcp2
pool.)

Felix



[1] 
https://lists.apache.org/thread.html/r598c0f654477372d112858af1c18bfc04008250156989647d883576f%40%3Cdev.commons.apache.org%3E





On Sat, Aug 29, 2020 at 6:50 PM Phil Steitz  wrote:


On 8/27/20 2:47 AM, Gokhan Akgul wrote:

Hi ,

I have been facing the deadlock issue for the last 2 months  about
JDBCPoolCleaner Thread .

Following config set in context.xml


  
url="jdbc:mysql://adress:3306/db?useUnicode=truecharacterEncoding=latin5characterResultSet=latin5zeroDateTimeBehavior=convertToNullautoReconnect=trueinteractiveClient=true"

   username="user"
   password="pass"
   initialSize="10"
   maxActive="30"
   maxIdle="15"
   minIdle="10"
   maxWait="3"
   timeBetweenEvictionRunsMillis="5000"
   minEvictableIdleTimeMillis="6"
   removeAbandonedTimeout="600"
   removeAbandoned="true"
   logAbandoned="false"
   testWhileIdle="true"
   testOnBorrow="true"
   testOnReturn="false"
   validationQuery="/* ping */ SELECT 1"
   validationInterval="3"
   jmxEnabled="true"


  
jdbcInterceptors="ConnectionState;StatementFinalizer;ResetAbandonedTimer;SlowQueryReport"

  />



Thread dump

Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 state=BLOCKED
  - waiting to lock <0x57dcb0b7> (a

com.mysql.jdbc.JDBC4PreparedStatement)

   owned by http-nio-8080-exec-8 id=25
  at

com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078)

  at

com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584)

  at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364)
  at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556)
  at

org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:388)

  at

org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.java:618)

  at

org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java:612)

  at

org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:569)

  at

org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPool.java:999)

  at

org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1468)

  at java.util.TimerThread.mainLoop(Timer.java:555)
  at java.util.TimerThread.run(Timer.java:505)

  Locked synchronizers: count = 1
-

java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868


http-nio-8080-exec-8 id=25 state=BLOCKED
  - waiting to lock <0x42de9995> (a com.mysql.jdbc.JDBC4Connection)
   owned by Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16
  at

com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl.java:5435)

  at

com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaData.java:3269)

  at com.mysql.jdbc.DatabaseMetaData.(DatabaseMetaData.java:675)
  at

com.mysql.jdbc.JDBC4DatabaseMetaData.(JDBC4DatabaseMetaData.java:39)

  at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown

Source)

  at

sun.reflect.Dele

Re: Tomcat JDBC Pool Cleaner Deadlock Problem

2020-08-29 Thread Phil Steitz



On 8/27/20 2:47 AM, Gokhan Akgul wrote:

Hi ,

I have been facing the deadlock issue for the last 2 months  about
JDBCPoolCleaner Thread .

Following config set in context.xml





Thread dump

Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 state=BLOCKED
 - waiting to lock <0x57dcb0b7> (a com.mysql.jdbc.JDBC4PreparedStatement)
  owned by http-nio-8080-exec-8 id=25
 at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078)
 at 
com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584)
 at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364)
 at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556)
 at 
org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:388)
 at 
org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.java:618)
 at 
org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java:612)
 at 
org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:569)
 at 
org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPool.java:999)
 at 
org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1468)
 at java.util.TimerThread.mainLoop(Timer.java:555)
 at java.util.TimerThread.run(Timer.java:505)

 Locked synchronizers: count = 1
   - java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868



http-nio-8080-exec-8 id=25 state=BLOCKED
 - waiting to lock <0x42de9995> (a com.mysql.jdbc.JDBC4Connection)
  owned by Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16
 at 
com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl.java:5435)
 at 
com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaData.java:3269)
 at com.mysql.jdbc.DatabaseMetaData.(DatabaseMetaData.java:675)
 at 
com.mysql.jdbc.JDBC4DatabaseMetaData.(JDBC4DatabaseMetaData.java:39)
 at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source)
 at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
 at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
 at com.mysql.jdbc.DatabaseMetaData.getInstance(DatabaseMetaData.java:657)
 at com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3064)
 at com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3056)
 at 
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2315)
 at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at 
org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210)
 at com.sun.proxy.$Proxy44.executeQuery(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at 
org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210)
 at com.sun.proxy.$Proxy44.executeQuery(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at 
org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
 at com.sun.proxy.$Proxy44.executeQuery(Unknown Source)
 at 
org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
 at org.hibernate.loader.Loader.getResultSet(Loader.java:1953)
 at org.hibernate.loader.Loader.doQuery(Loader.java:802)
 at 
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
 at org.hibernate.loader.Loader.doList(Loader.java:2542)
 at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
 at org.hibernate.loader.Loader.list(Loader.java:2271)
 at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:459)
 at 
org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:365)
 at 
org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
 at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268)
 at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
 at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:246)

 ..

 at 
com.gg.healthcheck.servlet.HealthCheckServlet.getServiceStatus(HealthCheckServlet.java:68)
 at 
com.gg.healthcheck.servlet.HealthCheckServlet.doGet(HealthCheckServlet.java:45)
 at 

Re: Reloading JNDI

2020-08-03 Thread Phil Steitz



On 7/24/20 10:46 AM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

All,

I have a JNDI  which is a JDBC DataSource. It is set to
singleton="true" via defaults (not explicitly set).

The JDBC Connections in this DataSource pool (using dbcp2 as provided
by Tomcat) have TLS configuration including client certificates,
trusted server certificates, etc.

I'd like to be able to "bounce" the DataSource so that each Connection
is re-established in order to pick-up any new TLS configuration --
specifically, reloading the key store and trust store for the connection
.

What's the best way to do that?


Do you want the connections that are checked out to clients to be 
force-closed, so they will get exceptions when they try to use them, or 
are you OK waiting until they get returned to the pool?


If you are OK being kind to clients, there is an enhancement request 
pending for DBCP (https://issues.apache.org/jira/browse/DBCP-559) 
 that might do what you 
want.  I  have not tested this, but I suspect that BDS closed followed 
by "open" (new method proposed, to be JMX-exposed) would cause idle 
connections to be closed immediately, checked out to be closed when they 
come back and new ones to be created in a new pool created by the open 
request.


If you really need to kill them all immediately, that would (as Mark 
said) require an invalidateAll or somesuch method added to commons pool, 
then exposed by DBCP as maybe closeAll.


Phil



It looks like Tomcat will automatically register a JMX bean for the
DataSource under
/Catalina:DataSource/[host]/[webapp]/javax.sql.DataSource/[dataSourceNam
e].
That JMX Bean has a "close" method that can be called. Calling "close"
seems to kill the DataSource, but a webapp reload can resurrect it.

There is also an "evict" method which takes zero arguments; that
appears to call the DBCP2 eviction process which removes "old" pool
members, not all.

I wasn't able to find anything else via the JMX method.

The class of the DataSource coming back from the JNDI lookup is
org.apache.tomcat.dbcp.dbcp2.BasicDataSource. It looks like I might be
able to downcast the object and call some other methods to stop /
restart it, but that seems fragile.

BasicDataSource.getConnectionPool is protected, but could be
forced-open via reflection without a SecurityManager (yuck). Then
GenericObjectPool.clear() could be called, but that only affects idle
connections in the pool and you can't be sue you've evicted all of them.

Any suggestions, other than reloading the context? I'm looking for a
zero-downtime solution.

Thanks,
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8bHoUACgkQHPApP6U8
pFgfqhAAol3JNOn/PIBZ0jD53N/jfIYjycF9NhHZ3EK7wMrB/clAl6rOVvR72C0E
ljt7Q2zx69+tTezd6yOqx5KejCU6oF9bVBR1NDdwAf1QPG50m+B35k6S2Aj6o3Ge
+5RNMS2OVDYHKi5nIxS5N1y8LtVcWBbo1LQN+QTjV6e5nEztg+fLct+HtDhwNYIv
KwRDiDeztR/ulxKrDEHpuJzUHmIa+c+hD6teYLLPtEcOD11fXXO/VTtO3ZL37qCn
JhEjb5ysoqHuIUH82CjzBghsEliZPczue5u7eaGhccnXR9y9WMM59yIf2jd3AH2L
2RVtw1/txdo3YmUGLJUi5IFQkQ8C8d13dSKT7I/KqO9NFywkiIPp7kimHmLdxvK5
vGJYFvZ6GV22oHWT0OaytaqbL1v4lf58k3frtt3bLRV7+2l78XUmsGsVksBr+3E+
7S0nFw7Cc7vvDJP0euvHlZEEoK+EsnTtDLd7EoFLU6Mct2U58NT1nmp5Aepnia8/
oKSAd1SaHiPCZeqHadQl8N4dyTuRKtwbH1Hlv9+o7hnyISLcwf5Nx9PHjgVn9BEZ
aOqpC5C0JxpGLXKDt4wTwjiIXdGo8c2T7pSV/pP82GnMhWAyHvGV1DPNHTvdoEKW
Y4Z/azWfNdKPbKxHcOgu3Lfx2oDFpDn40Yx5SK823ERn/hNUTG4=
=KgAg
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org





Re: Trying to chase down "too many connection" problems with DB

2018-03-28 Thread Phil Steitz
On 3/28/18 11:29 AM, Christopher Schultz wrote:
> Phil,
>
> On 3/27/18 1:03 PM, Phil Steitz wrote:
> > On 3/26/18 10:28 AM, Christopher Schultz wrote:
> >> Shawn,
> >>
> >> On 3/25/18 12:17 AM, Shawn Heisey wrote:
> >>> On 3/24/2018 5:04 PM, Mark Thomas wrote:
> >>>> Regarding your configuration:  >>>> auth="Container"
> >>>> factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory"
> >>>> driverClassName="com.mysql.jdbc.Driver"
> >>>> type="javax.sql.DataSource" maxActive="60" maxIdle="10"
> >>>> maxWait="3" removeAbandoned="true"
> >>>> removeAbandonedTimeout="30" username="REDACTED"
> >>>> password="REDACTED" testOnBorrow="true"
> >>>> validationQuery="select 1"
> >>>>
> >>
> url="jdbc:mysql://REDACTED.REDACTED.com:3306/REDACTED?autoReconnect=t
> >>
> >>
> ruezeroDateTimeBehavior=round"
> >>>>
> >>>>
> >>>>
> >> />
> >>>>
> >>>> Generally, that looks OK but I'd strongly recommend that you
> >>>> use "autoReconnect=false" in the URL. autoReconnect is known
> >>>> to be problematic with connection pools.
> >>>>
> >>>> The removeAbandonedTimeout looks low but if all the queries
> >>>> are expected to be well under 30s then that should be OK.
> >>
> >>> Somehow I did not see this part of your email at all when I
> >>> read it the first time.  I just noticed it.  My previous reply
> >>> probably has you scratching your head a little bit.  I'm sorry
> >>> about that!
> >>
> >>> The timeout of 30 seconds is EXTREMELY low.  And if it were
> >>> being honored, we would have customers lining up outside the
> >>> office with pitchforks.  The webapp has reporting ability for
> >>> users with elevated privileges, and a lot of those reports take
> >>> a minute or two to run, sometimes even longer.  So if the pool
> >>> were killing connections after 30 seconds, the reporting
> >>> mechanism would be failing a LOT.  If you check the *planned*
> >>> configuration, you'll see that I have increased this timeout to
> >>> 3600.  I wanted to make it 900, but some of the developers are
> >>> worried that 900 is too aggressive.
> >>
> >> The pool doesn't kill abandoned connections. It simply removes
> >> them from the pool. Otherwise, you're right: you'd have torches
> >> and pitchforks everywhere.
>
> > Not exactly, if what you are using is the DBCP pool.  To see the
> > details of what is going on, look at the removeAbandoned code in
> > DBCP's AbandonedObjectPool.  It calls
> > o.a.c.pool.GenericObjectPool#invalidateObject, which calls
> > o.a.c.dbco.PoolableConnectionFactory#destroyObject to close the
> > connection.  If an exception occurs, it is swallowed by
> > removeAbandoned, but it dumps a stack trace.
>
> Is this DBCP, or DBCP2, or both?
>
> OP is running Tomcat 7.0.x. which uses DBCP(1.x)

Above references are to the 1.x versions.  In 2.x, the
AbandonedObjectPool was moved into Commons pool, but the behavior is
basically the same - connections marked abandoned get invalidated by
the pool and destroyed by the factory.  Exceptions on close are
swallowed with stack traces dumped to stderr.

Phil
>
> -chris
> >
-
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For
additional commands, e-mail: users-h...@tomcat.apache.org > >


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Trying to chase down "too many connection" problems with DB

2018-03-28 Thread Phil Steitz
On 3/27/18 5:41 PM, Shawn Heisey wrote:
> On 3/27/2018 11:03 AM, Phil Steitz wrote:
>> Not exactly, if what you are using is the DBCP pool.  To see the
> The factory in use right now is
> "org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory".  Information
> gathered previously in this thread told me that this is DBCP code,
> repackaged into the tomcat package space to avoid conflicting with the
> official commons-dbcp package.  If this is effectively unmodified DBCP,
> then what you wrote below likely applies.

As Mark said above, based on your version of tomcat, it is exactly
dbcp 1.4, pool 1.5.7.
>
>> details of what is going on, look at the removeAbandoned code in
>> DBCP's AbandonedObjectPool.  It calls
>> o.a.c.pool.GenericObjectPool#invalidateObject, which calls
>> o.a.c.dbco.PoolableConnectionFactory#destroyObject to close the
>> connection.  If an exception occurs, it is swallowed by
>> removeAbandoned, but it dumps a stack trace.
>>
>> So connections should in fact be closed if they are detected as
>> abandoned.  As I said on commons-user, in your setup, that won't
>> happen unless borrows are attempted when there are 57+ open
>> connections.  The removeAbandoned method is called *only* by
>> borrowObject in DBCP 1.x, with this test:
> It would not surprise me to learn that on the pool with maxActive=60,
> the pool is managing less than 57 connections.  Until I can get some
> logging in place on production to show me the acive and idle connection
> counts, I do not know what's actually happening.
>
> If I can successfully get a config using
> "org.apache.tomcat.jdbc.pool.DataSourceFactory" operational (which seems
> to be failing right now with "too many connections" on startup), is it
> true that this pool does not require maxActive-3 connections before
> abandoned removal kicks in?  The description for
> abandonWhenPercentageFull in the tomcat documentation implies that this
> is the case.
>
> So now I have one person telling me that removeAbandoned DOES close
> connections, and another saying that it does NOT close connections.  Is
> there a conflict here, or are both of these statements correct for
> different pool implementations?  Keep in mind that the current config
> uses "org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory"

Using that one, you get the dbcp / pool combo above and abandoned
connection removal does try to close connections when they are
marked abandoned.  As I said, if the close fails, the failure should
result in a stack trace written to stderr.  There is a unit test in
dbcp 1.4, TestAbandonedBasicDataSource#testAbandonedClose that
verifies this behavior.

Phil

>  and that I
> would like to try a new config with
> "org.apache.tomcat.jdbc.pool.DataSourceFactory" ... unless there is a
> good reason to NOT use that factory.  Which, by the way, is the factory
> that tomcat's documentation SAYS to use.
>
> The new config with the new factory fails to start on our staging
> server, but aside from seeing "too many connection" exceptions in the
> catalina log, I have no concrete information about what happened.  I
> didn't do the restart, and wasn't watching the system when it happened. 
> Switching back to the old config fixed the tomcat startup.  What the
> developer described to me shouldn't have been possible.  I need to do
> the restart myself and watch the system.
>
> I've come up with another question, related to this, but headed in a
> different direction.  I'll write a new email to the list for that.
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Trying to chase down "too many connection" problems with DB

2018-03-27 Thread Phil Steitz
On 3/26/18 10:28 AM, Christopher Schultz wrote:
> Shawn,
>
> On 3/25/18 12:17 AM, Shawn Heisey wrote:
> > On 3/24/2018 5:04 PM, Mark Thomas wrote:
> >> Regarding your configuration:  >> auth="Container"
> >> factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory"
> >> driverClassName="com.mysql.jdbc.Driver"
> >> type="javax.sql.DataSource" maxActive="60" maxIdle="10"
> >> maxWait="3" removeAbandoned="true"
> >> removeAbandonedTimeout="30" username="REDACTED"
> >> password="REDACTED" testOnBorrow="true" validationQuery="select
> >> 1"
> >>
> url="jdbc:mysql://REDACTED.REDACTED.com:3306/REDACTED?autoReconnect=t
> ruezeroDateTimeBehavior=round"
> >>
> >>
> >>
> />
> >>
> >> Generally, that looks OK but I'd strongly recommend that you use
> >> "autoReconnect=false" in the URL. autoReconnect is known to be
> >> problematic with connection pools.
> >>
> >> The removeAbandonedTimeout looks low but if all the queries are
> >> expected to be well under 30s then that should be OK.
>
> > Somehow I did not see this part of your email at all when I read it
> > the first time.  I just noticed it.  My previous reply probably has
> > you scratching your head a little bit.  I'm sorry about that!
>
> > The timeout of 30 seconds is EXTREMELY low.  And if it were being
> > honored, we would have customers lining up outside the office with
> > pitchforks.  The webapp has reporting ability for users with
> > elevated privileges, and a lot of those reports take a minute or
> > two to run, sometimes even longer.  So if the pool were killing
> > connections after 30 seconds, the reporting mechanism would be
> > failing a LOT.  If you check the *planned* configuration, you'll
> > see that I have increased this timeout to 3600.  I wanted to make
> > it 900, but some of the developers are worried that 900 is too
> > aggressive.
>
> The pool doesn't kill abandoned connections. It simply removes them
> from the pool. Otherwise, you're right: you'd have torches and
> pitchforks everywhere.

Not exactly, if what you are using is the DBCP pool.  To see the
details of what is going on, look at the removeAbandoned code in
DBCP's AbandonedObjectPool.  It calls
o.a.c.pool.GenericObjectPool#invalidateObject, which calls
o.a.c.dbco.PoolableConnectionFactory#destroyObject to close the
connection.  If an exception occurs, it is swallowed by
removeAbandoned, but it dumps a stack trace.

So connections should in fact be closed if they are detected as
abandoned.  As I said on commons-user, in your setup, that won't
happen unless borrows are attempted when there are 57+ open
connections.  The removeAbandoned method is called *only* by
borrowObject in DBCP 1.x, with this test:

if (config != null
    && config.getRemoveAbandoned()
    && (getNumIdle() < 2)
    && (getNumActive() > getMaxActive() - 3) ) {
    removeAbandoned();
 }

Another thing you should do is to set logAbandoned to true so you
get a stack trace if / when abandoned connections are detected and
close.

Phil


>
> Sounds like you need to raise that timeout by a LOT.
>
> Also, set logAbandoned="true" and you'll get a helpful message every
> time a connection is considered abandoned and you'll find out if you
> have a connection leak (as opposed to simply a too-short "abandoned"
> setting).
>
> > Because there are no pitchforks, I know that abandoned connection
> > removal is NOT happening, even though it IS configured.
>
> It's probably happening, just not meeting your expectations. Those
> abandoned connections will pretty much live forever, no longer being
> managed by the pool and yet still counting as being used by the
> server. Maybe lower your idle-timeout on the server to help with this.
>
> Hope that helps,
> -chris
> >
-
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For
additional commands, e-mail: users-h...@tomcat.apache.org > >


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: /.well-known Hidden directory url returns 404

2017-05-02 Thread Phil Steitz
On 5/1/17 12:11 PM, Ian Brown wrote:
> I am trying to https/SSL enable my tomcat application server and have a 
> problem when I request verification from the CA.
> Let's Encrypt requires the certificate request to be placed in 
> mydomain.tld/.well-known/acme-challenge/ which they query to check that I 
> control the site.
> Tomcat does not appear to handle hidden directories correctly. There as 
> several on-line references to Tomcat being an issue, but I have yet to find a 
> Tomcat solution.
> (Other than to front tend Tomcat with the Apache httpd server, but I would 
> like to find a solution that is pure tomcat.)
> I started with tomcat-7.0.42 Centos 6-8.el6 jdk1.7.0_25,  then 
> upgraded to tomcat-8.5.14 Centos 6-8.el6jdk1.8.0_131, same problem.
>
> The hidden directory problem manifest in two ways. 
> 1. If I create a site/app with th directory /.well-known/ Tomcat creates two 
> contexts where there should be one, one for my app and another for 
> /.well-known (i.e. a sub directory of the app)
> 2. If I don't create a /.well-known/ directory, but try and do a urlrewrite 
> from /.well-known/ to say /well-known/ it still sees the url as trying to 
> access a separte context /.well-known/ 
> and does not rewrite it as expected.
>
> Request-dumper shows ( some lines removed for clarity)
>
> requestURI=/.well-known/acme-challenge/test.html
> contextPath=/.well-known
> serverName=mydomain.tld
> serverPort=80
> servletPath=/acme-challenge/test.html
> status=404
>
> The above fails if /.well-known/acme-challenge/test.html exists or not since 
> it is looking in the wrong context path.
>
> Contrasts with a correctly served (not hidden) page.
>
> requestURI=/stats/index.html
> contextPath=
> header=host=www.mydomain.tld
> contextPath=
> serverName=www.mydomain.tld
> serverPort=80
> servletPath=/stats/index.html
> status=200
>
> Has anyone seen an solution to this issue? Any suggestions?
> Thanks for your consideration, 
> ian
>

Hi Ian,

Hi Ian,

I worked around this for tomcat by using the "--standalone" option
for certbot.  That just requires that you have port 80 and 443
(temporarily) open for all while it does the verification.   My
Ansible playbook uses

certbot certonly --noninteractive --agree-tos --standalone --email
{{certbot_admin_email}} -d {{inventory_hostname}}

then

openssl pkcs12 -inkey {{certbot_output_dir}}/privkey.pem -in
{{certbot_output_dir}}/fullchain.pem -export -out
{{tomcat_keystore_dir}}/vlibrary.pfx -password pass:{{key_pwd}}

I agree it would be great to have an auto-install certbot setup for
tomcat, though.

Phil


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: LogAbandoned Stack Trace?

2017-01-10 Thread Phil Steitz
On 1/10/17 12:48 PM, Christopher Schultz wrote:
> Phil,
>
> On 1/8/17 5:41 PM, Phil Steitz wrote:
> > On 1/6/17 3:44 PM, Jerry Malcolm wrote:
> >> On 1/6/2017 4:30 PM, Christopher Schultz wrote:
> >>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
> >>>
> >>> Jerry,
> >>>
> >>> On 1/6/17 10:35 AM, Jerry Malcolm wrote:
> >>>> I'm getting "too many connections" errors.
> >>> Where?
> >>>
> >>> Can you provide an exact error message and, better yet, a
> >>> stack trace?
> >>>
> >>>> I'm pretty sure I am configured with enough connections that
> >>>> I shouldn't run out.  So I'm assuming I'm leaving some
> >>>> connections open.
> >>> That's a good assumption.
> >>>
> >>>> I have LogAbandoned="true" in my jdbc resource statements.
> >>>> The doc says TC will log a stack trace of abandoned
> >>>> connections.  But I don't see any stack traces.  Would they
> >>>> be in stderr, stdout, catalina log? Or is it that I'm
> >>>> actually not getting any abandoned?
> >>> Which db connection pool are you using? Standard (DBCP-based)
> >>> or tomcat-pool? A full (sanitized)  configuration
> >>> would help.
> >>>
> >>> - -chris
> >>>
> >> Chris, Stack trace follows.  It looks like it may be mySQL
> >> that's rejecting the connection.  But even if that's the case,
> >> it's probably because I'm not closing some connections, which
> >> should still generate a logAbandoned stack trace, correct?  I
> >> believe I'm using dbcp.  Not doing anything fancy... Just
> >> defining data source resources in the context file:
> >>
> >>  >> name="jdbc/cis" auth="Container" type="javax.sql.DataSource"
> >> maxTotal="100" maxIdle="30" maxWaitMillis="1"
> >> removeAbandoned="true" removeAbandonedTimeout="60"
> >> logAbandoned="true" username="" password="xxx"
> >> driverClassName="com.mysql.jdbc.Driver"
> >> url="jdbc:mysql://localhost:3306/xx" />
> > In dbcp 2, the "removeAbandoned" config option was replaced by
> > removedAbondonedOnBorrow and removeAbandonedOnMaintenance.  You
> > need to set one of these to true the get abandoned connection
> > cleanup to work.  See [1].
>
> > |Phil
>
> > [1]
> > http://commons.apache.org/proper/commons-dbcp/configuration.html
>
> +1
>
> Jerry never said what version of Tomcat he was using... I was assuming
> a DBCP 1.x-based version given his configuration.

>From the stack trace, you can see dbcp2 in the package names.  I am
correct in assuming that tomcat does not kindly
s/removeAbandoned/removeAbandonedOnBorrow, right?

Phil
>
> -chris
> >
-
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For
additional commands, e-mail: users-h...@tomcat.apache.org > >



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: LogAbandoned Stack Trace?

2017-01-08 Thread Phil Steitz
On 1/6/17 3:44 PM, Jerry Malcolm wrote:
> On 1/6/2017 4:30 PM, Christopher Schultz wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA256
>>
>> Jerry,
>>
>> On 1/6/17 10:35 AM, Jerry Malcolm wrote:
>>> I'm getting "too many connections" errors.
>> Where?
>>
>> Can you provide an exact error message and, better yet, a stack
>> trace?
>>
>>> I'm pretty sure I am configured with enough connections that I
>>> shouldn't run out.  So I'm assuming I'm leaving some connections
>>> open.
>> That's a good assumption.
>>
>>> I have LogAbandoned="true" in my jdbc resource statements.  The
>>> doc says TC will log a stack trace of abandoned connections.  But I
>>> don't see any stack traces.  Would they be in stderr, stdout,
>>> catalina log? Or is it that I'm actually not getting any
>>> abandoned?
>> Which db connection pool are you using? Standard (DBCP-based) or
>> tomcat-pool? A full (sanitized)  configuration would help.
>>
>> - -chris
>>
> Chris,
> Stack trace follows.  It looks like it may be mySQL that's
> rejecting the connection.  But even if that's the case, it's
> probably because I'm not closing some connections, which should
> still generate a logAbandoned stack trace, correct?  I believe I'm
> using dbcp.  Not doing anything fancy... Just defining data source
> resources in the context file:
>
> name="jdbc/cis" auth="Container" type="javax.sql.DataSource"
> maxTotal="100" maxIdle="30" maxWaitMillis="1"
> removeAbandoned="true" removeAbandonedTimeout="60"
> logAbandoned="true" username="" password="xxx"
> driverClassName="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost:3306/xx" />
In dbcp 2, the "removeAbandoned" config option was replaced by
removedAbondonedOnBorrow and removeAbandonedOnMaintenance.  You need
to set one of these to true the get abandoned connection cleanup to
work.  See [1].

|Phil

[1] http://commons.apache.org/proper/commons-dbcp/configuration.html
|||
>
>
>
> java.sql.SQLException: Cannot create PoolableConnectionFactory
> (Data source rejected establishment of connection,  message from
> server: "Too many connections")
> at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2195)
> at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1945)
> at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
> at
> jwm.servletdb.ServletDBData.getInstance(ServletDBData.java:139)
> at
> jwm.servletdb.ServletDBData.getInstance(ServletDBData.java:93)
> at
> jwm.servletdb.ServletDBData.getInstance(ServletDBData.java:62)
> at
> org.apache.jsp.jsp.login.login_005fv2_jsp._jspService(login_005fv2_jsp.java:450)
> at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
>
> at
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
> at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:721)
> at
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:466)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:391)
> at
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:318)
> at
> org.apache.catalina.authenticator.FormAuthenticator.forwardToLoginPage(FormAuthenticator.java:411)
> at
> org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:256)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
> at
> org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:291)
> at
> 

Re: tomcat 7 connection pool validation interval

2016-07-03 Thread Phil Steitz
On 7/1/16 11:35 PM, Nir Dweck wrote:
> -Original Message-
> From: Phil Steitz [mailto:phil.ste...@gmail.com] 
> Sent: Friday, July 01, 2016 9:16 PM
> To: users@tomcat.apache.org
> Subject: Re: tomcat 7 connection pool validation interval
>
> On 7/1/16 7:14 AM, Nir Dweck wrote:
>> -Original Message-
>> From: Phil Steitz [mailto:phil.ste...@gmail.com]
>> Sent: Thursday, June 30, 2016 6:57 PM
>> To: Tomcat Users List
>> Subject: Re: tomcat 7 connection pool validation interval
>>
>>
>>
>>>>> On Jun 30, 2016, at 9:32 AM, Nir Dweck <n...@vasco-de.com> wrote:
>>>>>
>>>>> I am using tomcat connection  pool (tomcat 7) in my application (java 
>>>>> application) to connect to a remote Oracle DB on an Azure machine. My 
>>>>> connection pool configuration is as follow:
>>>>>
>>>>>PoolProperties p = new PoolProperties();
>>>>>p.setUrl(connString);
>>>>>p.setUsername(user);
>>>>>p.setPassword(password);
>>>>>p.setDriverClassName("oracle.jdbc.OracleDriver");
>>>>>p.setValidationQuery("SELECT 1 from dual");
>>>>>p.setValidationInterval(1 * 6/2);
>>>>>p.setTimeBetweenEvictionRunsMillis(1 * 6/2);
>>>>>p.setTestOnBorrow(true);
>>>>>p.setTestWhileIdle(true);
>>>>>p.setMinIdle(10);
>>>>>p.setInitialSize(10);
>>>>> However looking at the capture file, I see that the connections are not 
>>>>> checked every 30 seconds as I expected. One connection is checked 
>>>>> correctly, the others are checked after 3 times the configuration (180 
>>>>> second). then again only some of the connections are checked and after a 
>>>>> while it seems to stable on twice the configuration period (every 60 
>>>>> seconds).
>>>>>
>>>>> I tested it with different configured time and different pool size, all 
>>>>> had a instability period in which each time only part of the connections 
>>>>> were checked and eventually it stabled on checks of all the connections 
>>>>> every twice the configured time.
>>>>> What am I missing?
>>>>> Thanks,
>>>> Most likely the connections not being tested are checked out to clients 
>>>> when the pool maintenance runs.  Assuming you are using the default DBCP 
>>>> pool, >>only idle connections (meaning waiting in the pool)  are tested by 
>>>> the maintenance thread.
>>>> Phil
>>> No, all connection are idle, but still looking at wireShark the validation 
>>> query is sent every twice the TimeBetweenEvictionRunsMillis.
>> I am sorry I should have been more clear.  The term, "idle" is a little 
>> confusing here.  I should have said connections are only checked when they 
>> are not >checked out to client threads.  A connection can be "idle" from the 
>> DB standpoint but still checked out to a client.  A connection that is 
>> checked out to a >client will not be tested by the pool maintenance thread.  
>> Are there clients using the pool while you are running this test?
>> I notice now a parameter not supported by DBCP, so I guess you must be using 
>> the tomcat-jdbc pool.  It looks to me like it defines "idle" the same way 
>> DBCP does - in other words above statements are true for that pool too.
>> It is also possible that setting validationInterval to a value <= 
>> timeBetweenEvictionRuns may cause some validations to be suppressed, since 
>> the former is a >max frequency.
>> Phil
> Hi Phil,
> I understood you correctly, none of the connections are held by any client. 
> All are idle in the pool. I tried also setting timeBetweenEvictionRuns to 
> half validationInterval, but still, the best I get is that the connections 
> are validated in half the frequency I requested (every 2 * 
> validationInterval). And yes I am using tomcat-jdbc pool.

Sorry I misunderstood you.  Try going the other way - making sure
that validationInterval >> timeBetweenEvictionRuns, or get rid of
the validationInterval parameter altogether.  The validationInterval
setting is designed to *suppress* excess validation - i.e., to make
sure that connections are not validated too frequently.  See the
description of this parameter in [1].

Phil
[1]
https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Tomcat_JDBC_Enhanced_Attributes

>  
> Nir
>>>  
>> Ni

Re: tomcat 7 connection pool validation interval

2016-07-01 Thread Phil Steitz
On 7/1/16 7:14 AM, Nir Dweck wrote:
> -Original Message-
> From: Phil Steitz [mailto:phil.ste...@gmail.com] 
> Sent: Thursday, June 30, 2016 6:57 PM
> To: Tomcat Users List
> Subject: Re: tomcat 7 connection pool validation interval
>
>
>
>>> On Jun 30, 2016, at 9:32 AM, Nir Dweck <n...@vasco-de.com> wrote:
>>>
>>> I am using tomcat connection  pool (tomcat 7) in my application (java 
>>> application) to connect to a remote Oracle DB on an Azure machine. My 
>>> connection pool configuration is as follow:
>>>
>>>PoolProperties p = new PoolProperties();
>>>p.setUrl(connString);
>>>p.setUsername(user);
>>>p.setPassword(password);
>>>p.setDriverClassName("oracle.jdbc.OracleDriver");
>>>p.setValidationQuery("SELECT 1 from dual");
>>>p.setValidationInterval(1 * 6/2);
>>>p.setTimeBetweenEvictionRunsMillis(1 * 6/2);
>>>p.setTestOnBorrow(true);
>>>p.setTestWhileIdle(true);
>>>p.setMinIdle(10);
>>>p.setInitialSize(10);
>>> However looking at the capture file, I see that the connections are not 
>>> checked every 30 seconds as I expected. One connection is checked 
>>> correctly, the others are checked after 3 times the configuration (180 
>>> second). then again only some of the connections are checked and after a 
>>> while it seems to stable on twice the configuration period (every 60 
>>> seconds).
>>>
>>> I tested it with different configured time and different pool size, all had 
>>> a instability period in which each time only part of the connections were 
>>> checked and eventually it stabled on checks of all the connections every 
>>> twice the configured time.
>>> What am I missing?
>>> Thanks,
>> Most likely the connections not being tested are checked out to clients when 
>> the pool maintenance runs.  Assuming you are using the default DBCP pool, 
>> >only idle connections (meaning waiting in the pool)  are tested by the 
>> maintenance thread.
>> Phil
> No, all connection are idle, but still looking at wireShark the validation 
> query is sent every twice the TimeBetweenEvictionRunsMillis.

I am sorry I should have been more clear.  The term, "idle" is a
little confusing here.  I should have said connections are only
checked when they are not checked out to client threads.  A
connection can be "idle" from the DB standpoint but still checked
out to a client.  A connection that is checked out to a client will
not be tested by the pool maintenance thread.  Are there clients
using the pool while you are running this test?

I notice now a parameter not supported by DBCP, so I guess you must
be using the tomcat-jdbc pool.  It looks to me like it defines
"idle" the same way DBCP does - in other words above statements are
true for that pool too.

It is also possible that setting validationInterval to a value <=
timeBetweenEvictionRuns may cause some validations to be suppressed,
since the former is a max frequency.

Phil
>  
>
> Nir
>
>> Nir
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: tomcat 7 connection pool validation interval

2016-06-30 Thread Phil Steitz


> On Jun 30, 2016, at 9:32 AM, Nir Dweck  wrote:
> 
> I am using tomcat connection  pool (tomcat 7) in my application (java 
> application) to connect to a remote Oracle DB on an Azure machine. My 
> connection pool configuration is as follow:
> 
>PoolProperties p = new PoolProperties();
>p.setUrl(connString);
>p.setUsername(user);
>p.setPassword(password);
>p.setDriverClassName("oracle.jdbc.OracleDriver");
>p.setValidationQuery("SELECT 1 from dual");
>p.setValidationInterval(1 * 6/2);
>p.setTimeBetweenEvictionRunsMillis(1 * 6/2);
>p.setTestOnBorrow(true);
>p.setTestWhileIdle(true);
>p.setMinIdle(10);
>p.setInitialSize(10);
> However looking at the capture file, I see that the connections are not 
> checked every 30 seconds as I expected. One connection is checked correctly, 
> the others are checked after 3 times the configuration (180 second). then 
> again only some of the connections are checked and after a while it seems to 
> stable on twice the configuration period (every 60 seconds).
> 
> I tested it with different configured time and different pool size, all had a 
> instability period in which each time only part of the connections were 
> checked and eventually it stabled on checks of all the connections every 
> twice the configured time.
> What am I missing?
> Thanks,

Most likely the connections not being tested are checked out to clients when 
the pool maintenance runs.  Assuming you are using the default DBCP pool, only 
idle connections (meaning waiting in the pool)  are tested by the maintenance 
thread.

Phil
> 
> Nir

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: AW: AW: AW: Tomcat 6, DB2 Driver Problems

2015-10-30 Thread Phil Steitz
On 10/29/15 7:14 AM, Christopher Schultz wrote:
> Simon,
>
> On 10/29/15 4:28 AM, simone.rodenbach@devk.de wrote:
>> Thx,
>>
>> I hope this information helps: (The 
>> org.apache.commons.pool.impl.GenericObjectPool starts a timer ... )
>>
>> ava.util.TimerThread @ 0xc0772288
>>|Timer-0|  128 |   384 
>> |org.apache.catalina.loader.WebappClassLoader @ 0xc04bed30|true
>> |- at java.lang.Object.wait(J)V (Native Method)  
>> |   |  |   | 
>> |
>> |- at java.util.TimerThread.mainLoop()V (Timer.java:552) 
>> |   |  |   | 
>> |
>> |- at java.util.TimerThread.run()V (Timer.java:505)  
>> |   |  |   | 
>> |
>> |  '-  java.util.TimerThread @ 0xc0772288  Timer-0 Thread 
>> |   |  128 |   384 | 
>> |
>> | |-  class java.util.TimerThread @ 0xc0cd5080 System Class   
>> |   |0 | 0 | 
>> |
>> | |- group java.lang.ThreadGroup @ 0xc048b400  main  
>> |   |   48 |   208 | 
>> |
>> | |- contextClassLoader org.apache.catalina.loader.WebappClassLoader @ 
>> 0xc04bed30|   |  200 | 1.314.384 |   
>>   |
>> | |- , queue java.util.TaskQueue @ 0xc0758a80 Busy Monitor   
>> |   |   24 | 1.528 | 
>> |
>> | |-  java.util.TimerThread @ 0xc0772288  Timer-0 Thread 
>> |   |  128 |   384 | 
>> |
>> | |- name char[7] @ 0xc0772408  Timer-0  
>> |   |   32 |32 | 
>> |
>> | |- inheritedAccessControlContext java.security.AccessControlContext @ 
>> 0xc0772428   |   |   40 |   104 |
>>  |
>> | |- inheritableThreadLocals java.lang.ThreadLocal$ThreadLocalMap @ 
>> 0xc0772490   |   |   24 |   104 |
>>  |
>> | |- blockerLock java.lang.Object @ 0xc07724f8   
>> |   |   16 |16 | 
>> |
>> | |-  org.apache.commons.pool.impl.GenericObjectPool$Evictor 
>> @ 0xeefe76d0|   |   40 |40 | 
>> |
>> | |-  java.lang.Object @ 0xeefe76f8  
>> |   |   16 |16 | 
>> |
>> | '- Total: 11 entries   
>> |   |  |   | 
>> |
>> '- Total: 3 entries  
>> |   |  |   | 
>> |
>> 
> This was very difficult to interpret.
>
>> The stacktrace shows only:
>>
>> Timer-1
>>   at java.lang.Object.wait(J)V (Native Method)
>>   at java.util.TimerThread.mainLoop()V (Timer.java:552)
>>   at java.util.TimerThread.run()V (Timer.java:505)
> This was not, and it was about what I expected.
>
> Usually when a thread if blocked on a monitor, it will give you the hex
> address of the object being used as the monitor, and then you can do
> find out what object that is. In the case of the Timer, it might just be
> it's waiting on itself.
>
> Once you find out what that timer does when it wakes up, you'll probably
> find out who created the timer.
>
> If the timer thread truly is executing the GenericObjectEvictor, then
> there may be a bug in Tomcat. I don't see anything in the changelog that
> would explain this, but could you re-try your testing with Tomcat 6.0.44?

If it is a problem related to the Evictor, it is likely in commons
pool.  As Chris pointed out, the config specified the commons dbcp
datasource directly, rather than using the tomcat-provided
repackaged 

Re: [OT] Too Many Connections exceptions after moving to Tomcat 8

2015-08-04 Thread Phil Steitz
On 8/4/15 2:03 PM, Christopher Schultz wrote:
 Phil,

 On 8/1/15 9:34 PM, Phil Steitz wrote:
  On 8/1/15 6:14 PM, Christopher Schultz wrote:
  Phil,
 
  On 8/1/15 11:12 AM, Phil Steitz wrote:
  Sorry the docs are not that clear.  The problem is that the
  config properties work together to determine behavior and
  documenting them individually makes it hard to put the whole
  picture together. Improvement patches most welcome :)  In any
  case, here is how it works:
 
  Connections are evaluated for abandonment when they are out in
  circulation - checked out to clients.  If you have set
  timeBetweenEvictionRunsMillis to a positive value, pool
  maintenance runs every timeBetweenEvictionRunsMillis
  milliseconds. If you have removeAbandonedOnMaintenance set to
  true, each time maintenance runs the pool removes abandoned
  connections.  If you have removeAbandonedOnBorrow set to true,
  the pool removes abandoned connections if it is nearing
  depletion when a borrow request arrives.  In both cases, the
  pool looks at the statistics that it maintains on the list of
  all objects checked out by clients to determine which ones
  appear to be abandoned.  To appear abandoned means to be
  checked out by a client but not used for longer than the
  removeAbandonedTimeout.  For DBCP, used means the connection
  has been used.
 
  So when during all that does the abandoned connection get
  logged?

  The log message is written when the abandoned connection is closed
  by the pool.

 So if the client code never calls close(), does that mean the
 amandonment is never logged? IIRC, DBCP 1.x would log the stack trace
 as soon as removeAbandonedTimeout elapsed if the connection hadn't
 been returned.

The behavior is the same in 2.x as 1.x with the exception that in
2.x, pool maintenance can also trigger abandoned object removal.  In
1.x, it is necessary to attempt a borrow to get abandoned
connections to be removed.  Both 1.x and 2.x essentially remove
abandoned connections in batches, closing the connections and
logging the traces when the batches run.  In 1.x, the only way to
trigger an abandoned cleanup batch is to request a connection when
the pool is near depletion.  In 2.x, if removeAbandonedOnMaintenance
is true, an abandoned cleanup batch will run every
timeBetweenEvictionRunsMillis ms.

  (The pool bug was basically delaying the effective write; but the
  stack trace goes to the buffer when the close happens)
 
  If I have a pool with two connections in it, and the first one
  gets leaked (after, lets say, 10 seconds) but the second one is
  returned properly, and the application continues to borrow and
  return the second connection, what happens?

  That depends on the settings.  See below.
 
  Are you only checking connections for abandonment after they
  are returned and then when a client attempts to borrow that
  specific connection? Or, are all currently-borrowed connections
  checked for abandonment whenever any borrow operation is
  requested?

  It is the latter, assuming removeAbandonedOnBorrow is true.
  Removal on borrow also requires that the pool be near depletion
  when the borrow happens (see the javadoc for precise description
  of conditions).  There are two things that can trigger abandoned
  object removal - a borrow request and pool maintenance.  In each
  case, if the config setting to enable the check on that event is
  on, all checked out objects are examined.  The ones that appear
  abandoned are destroyed (in DBCP that means physical connections
  are closed).

 Why is there no option to dump a warning as soon as the timeout
 elapses? 

Ideas / patches welcome on how to do that efficiently.

 I suppose if the pool is being used slightly more than
 /never/, the abandoned notice will happen pretty quickly, but for
 testing purposes, it means that a single-user will have to perform
 some operation, wait, and then perform another operation in order to
 get the feedback about the abandonment. That's surprising behavior if
 you ask me.

See above.  If you set removeAbandonedOnMaintenance and
maxTimeBetweenEvictionRunsMillis, abandoned connection removal will
happen regularly.

Phil



 -chris

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org






-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Too Many Connections exceptions after moving to Tomcat 8

2015-08-01 Thread Phil Steitz
On 7/31/15 8:16 AM, Christopher Schultz wrote:
 Phil,

 On 7/30/15 8:05 PM, Phil Steitz wrote:
  On 7/30/15 9:03 AM, Christopher Schultz wrote:
  Jerry,
 
  On 7/29/15 3:25 PM, Jerry Malcolm wrote:
  Well, it appears that we are slowly getting to the bottom of
  this. But with every answer, I get a few more questions
 
  First, I installed the latest TC8 on my laptop, copied my
  server.conf and conf/catalina folder to it and started it up
  just to see what errors I got.  After changing out an obsolete
  listener, it came up.  I found all of the resource parm
  exceptions in stderr.  So that question is cleared up.  Thanks
  for clarifying where to find that.
 
  If you have an obsolete Listener, you probably copied your
  server.xml from Tomcat 7 to Tomcat 8 which, while being less of a
  disaster than with previous version-pairs, is not good practice.
 
  Instead, start with the stock server.xml that comes with your
  Tomcat version and modify it to suit. These days, you should
  pretty much only have to configure the Executor and Connector
  elements, unless you have a particularly exotic Host
  configuration.
 
  The site is a wedding vendor advertiser site that spans two
  major cities.  There is no user login.  Simply a very huge
  online catalog. I'm certain it's deployed only once.  Whether I
  need that many connections is a valid question.  As far as I
  know, I haven't hit the limit in normal operation until now.
  Could possibly reduce the count if I collect statistics.
 
  Our user load is roughly 250 concurrently logged-in users per
  Tomcat node, and we have maxTotal=20. I never get alarms about
  hitting that maximum. Your requirements may be different.
 
  I've been monitoring the production server logs all day
  watching to be sure connection pool doesn't dry up again.
  About an hour ago, there was a single huge dump in stdout of
  approx 2000 'logAbandoned' exceptions. They showed connections
  from 1am right after my last bounce of the server thru 1:35pm.
 
  It looks like your startup process (likely loading and caching
  stuff from the db on launch) is leaky. That can run-up your
  connection could quite quickly.
 
  The good news is with the stack trace on one of them I was able
  to see the bug causing the leak.
 
  Good.
  But why did it decide to wait over 12 hours accumulating
  abandoned connections before dumping them back in the pool?
 
  I was about to say the following, but markt says it might be a
  bug in DBCP.

  The bug is in Commons Pool (POOL-300).  It is not flushing its
  abandoned object log.  That means abandoned traces won't appear
  (in the default System.out configuration) until some have been
  accumulated.

 Thanks for the correction.

  I'll say it anyway:
 
  DBCP 2 looks like it only checks for abandoned connections on
  borrow so it might only log their abandonment when you see a
  flurry of connection-checkouts occurring, not when the
  connections are returned to the pool. DBCP 1 would complain
  pretty much immediately when the timeout was reached and the
  connection hadn't been returned.

  When DBCP checks for abandoned connections depends in its
  configuration properties.  There are two relevant properties:
  removeAbandonedOnBorrow and removeAbandonedOnMaintenance.

 Right. I think most people don't use the maintenance mode, so I was
 being sloppy. I haven't read the code, but the configuration options
 make it sound like the connection isn't checked until it's borrowed
 again from the pool, which could be a very long time after it would be
 expected to have been abandoned.

Sorry the docs are not that clear.  The problem is that the config
properties work together to determine behavior and documenting them
individually makes it hard to put the whole picture together. 
Improvement patches most welcome :)  In any case, here is how it works:

Connections are evaluated for abandonment when they are out in
circulation - checked out to clients.  If you have set
timeBetweenEvictionRunsMillis to a positive value, pool maintenance
runs every timeBetweenEvictionRunsMillis milliseconds.  If you have
removeAbandonedOnMaintenance set to true, each time maintenance runs
the pool removes abandoned connections.  If you have
removeAbandonedOnBorrow set to true, the pool removes abandoned
connections if it is nearing depletion when a borrow request
arrives.  In both cases, the pool looks at the statistics that it
maintains on the list of all objects checked out by clients to
determine which ones appear to be abandoned.  To appear abandoned
means to be checked out by a client but not used for longer than the
removeAbandonedTimeout.  For DBCP, used means the connection has
been used.

Phil


  I realize from now knowing the code bug that the leak is a
  slow drip that is continually leaking on a regular basis. But
  since that last 12-hour accumulated dump, the abandoning has
  returned to silence. Since leaks are occurring regularly and
  would be timing out

Re: [OT] Too Many Connections exceptions after moving to Tomcat 8

2015-08-01 Thread Phil Steitz
On 8/1/15 6:14 PM, Christopher Schultz wrote:
 Phil,

 On 8/1/15 11:12 AM, Phil Steitz wrote:
  Sorry the docs are not that clear.  The problem is that the config
  properties work together to determine behavior and documenting
  them individually makes it hard to put the whole picture together.
   Improvement patches most welcome :)  In any case, here is how it
  works:

  Connections are evaluated for abandonment when they are out in
  circulation - checked out to clients.  If you have set
  timeBetweenEvictionRunsMillis to a positive value, pool
  maintenance runs every timeBetweenEvictionRunsMillis milliseconds.
  If you have removeAbandonedOnMaintenance set to true, each time
  maintenance runs the pool removes abandoned connections.  If you
  have removeAbandonedOnBorrow set to true, the pool removes
  abandoned connections if it is nearing depletion when a borrow
  request arrives.  In both cases, the pool looks at the statistics
  that it maintains on the list of all objects checked out by clients
  to determine which ones appear to be abandoned.  To appear
  abandoned means to be checked out by a client but not used for
  longer than the removeAbandonedTimeout.  For DBCP, used means the
  connection has been used.

 So when during all that does the abandoned connection get logged?

The log message is written when the abandoned connection is closed
by the pool.  (The pool bug was basically delaying the effective
write; but the stack trace goes to the buffer when the close happens)

 If I have a pool with two connections in it, and the first one gets
 leaked (after, lets say, 10 seconds) but the second one is returned
 properly, and the application continues to borrow and return the
 second connection, what happens?

That depends on the settings.  See below.

 Are you only checking connections for abandonment after they are
 returned and then when a client attempts to borrow that specific
 connection? Or, are all currently-borrowed connections checked for
 abandonment whenever any borrow operation is requested?

It is the latter, assuming removeAbandonedOnBorrow is true.  Removal
on borrow also requires that the pool be near depletion when the
borrow happens (see the javadoc for precise description of
conditions).  There are two things that can trigger abandoned object
removal - a borrow request and pool maintenance.  In each case, if
the config setting to enable the check on that event is on, all
checked out objects are examined.  The ones that appear abandoned
are destroyed (in DBCP that means physical connections are closed).

Phil

 Thanks,
 -chris

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org






-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Too Many Connections exceptions after moving to Tomcat 8

2015-07-30 Thread Phil Steitz
On 7/30/15 9:03 AM, Christopher Schultz wrote:
 Jerry,

 On 7/29/15 3:25 PM, Jerry Malcolm wrote:
  Well, it appears that we are slowly getting to the bottom of this.
  But with every answer, I get a few more questions

  First, I installed the latest TC8 on my laptop, copied my
  server.conf and conf/catalina folder to it and started it up just
  to see what errors I got.  After changing out an obsolete listener,
  it came up.  I found all of the resource parm exceptions in
  stderr.  So that question is cleared up.  Thanks for clarifying
  where to find that.

 If you have an obsolete Listener, you probably copied your server.xml
 from Tomcat 7 to Tomcat 8 which, while being less of a disaster than
 with previous version-pairs, is not good practice.

 Instead, start with the stock server.xml that comes with your Tomcat
 version and modify it to suit. These days, you should pretty much only
 have to configure the Executor and Connector elements, unless you
 have a particularly exotic Host configuration.

  The site is a wedding vendor advertiser site that spans two major
  cities.  There is no user login.  Simply a very huge online
  catalog. I'm certain it's deployed only once.  Whether I need that
  many connections is a valid question.  As far as I know, I haven't
  hit the limit in normal operation until now.  Could possibly reduce
  the count if I collect statistics.

 Our user load is roughly 250 concurrently logged-in users per Tomcat
 node, and we have maxTotal=20. I never get alarms about hitting that
 maximum. Your requirements may be different.

  I've been monitoring the production server logs all day watching to
  be sure connection pool doesn't dry up again.  About an hour ago,
  there was a single huge dump in stdout of approx 2000
  'logAbandoned' exceptions. They showed connections from 1am right
  after my last bounce of the server thru 1:35pm.

 It looks like your startup process (likely loading and caching stuff
 from the db on launch) is leaky. That can run-up your connection could
 quite quickly.

  The good news is with the stack trace on one of them I was able to
  see the bug causing the leak.

 Good.
  But why did it decide to wait over 12 hours accumulating abandoned
  connections before dumping them back in the pool?

 I was about to say the following, but markt says it might be a bug in
 DBCP.

The bug is in Commons Pool (POOL-300).  It is not flushing its
abandoned object log.  That means abandoned traces won't appear (in
the default System.out configuration) until some have been
accumulated. 
 I'll say it anyway:

 DBCP 2 looks like it only checks for abandoned connections on borrow
 so it might only log their abandonment when you see a flurry of
 connection-checkouts occurring, not when the connections are returned
 to the pool. DBCP 1 would complain pretty much immediately when the
 timeout was reached and the connection hadn't been returned.

When DBCP checks for abandoned connections depends in its
configuration properties.  There are two relevant properties:
removeAbandonedOnBorrow and removeAbandonedOnMaintenance.

  I realize from now knowing the code bug that the leak is a slow
  drip that is continually leaking on a regular basis. But since that
  last 12-hour accumulated dump, the abandoning has returned to
  silence. Since leaks are occurring regularly and would be timing
  out regularly, shouldn't I see a similar 'slow drip' of
  logAbandoned entries in stdout instead of a big dump every 12
  hours?

  It's going to take a day or two to fix the leak, test, and deploy.

 For testing, set maxTotal=1. You'll find your leaks *very quickly*
 that way, because everything will come grinding to a halt when you try
 to fetch that second connection from the pool.

  If indeed abandoned connections are now correctly being returned to
  the pool, then I presume we are back to working the way it did on
  TC7. Still not sure why it started working now.  But I guess once I
  get the leak fixed and if TC8 is now configured to handle abandoned
  connections, I'm good.  Still would like to know about the
  mega-dump vs. trickle of abandoned connections being logged.

 You should be able to run in testing with an upgraded DBCP 2. You
 might have to build it from trunk, though. I'm not sure if you are
 okay with that, but it might help you with your testing.

The thing to swap out is Commons Pool.  There is a release VOTE in
progress now for an RC including a fix for POOL-300.

A workaround that should work is to get a reference to the
BasicDataSource instance, say, bds and do

bds.setAbandonedLogWriter(new PrintWriter(System.out, true));

before using the pool.

Not sure if this will work correctly to get the output properly
directed under tomcat; but it is worth a try.

Phil

 Glad you are rooting-out some problems with your code.

 I highly recommend the use of findbugs:
 http://findbugs.sourceforge.net/

 It automatically detects stuff like this in your code, as well as a
 

Re: Java stack trace, unable to connect to the database

2015-07-14 Thread Phil Steitz
On 7/14/15 1:11 PM, Christopher Schultz wrote:
 Laurence,

 On 7/14/15 2:32 PM, Laurence Cohen wrote:
  We are suddenly having a problem with our Tomcat WebApp connecting
  to the database.  Unfortunately I can't give you the whole thing
  because the error is on a network that is restricted, and I can't
  take anything off of it without writing it down and retyping it.  I
  figured I would give you what information I can, and see if anyone
  has any ideas.

  The first error before the stack trace is Error preloading the
  connection pool.

 Hmm. I don't see the string preloading anywhere in Tomcat 7. Can you
 ask the engineers if they can find that string in their own code?

DBCP will throw an SQLException with that error message if an
exception occurs when it is trying to preload the pool.

Phil


  The last error is FATAL Database is starting up.

 That's definitely not a Tomcat error message.

  Unfortunately we do not have access to the database server, but I
  can tell you it's Postgresql.  We are running Tomcat 7.054.  I'm
  not sure of the Postgresql version because it is not under our
  control.  The database admins say they see us connect for a couple
  of seconds and then we disconnect. This is on both of the webapps
  that we are running.

 Unfortunately, I think just about anything could be the problem. I
 might start by checking to see if the configuration for the connection
 pool might try to make more connections than Postgres is willing to
 allocate to the user (e.g. you are only allowed to open 10 connections
 but you are trying to open 15). In that case, the connection might
 fail and the pool (which is trying to fill itself) might report a
 failure to initialize.

 This may or may not have anything to do with Tomcat; it depends upon
 the pool you are using, how you are initializing it, etc.

 Try to get some more information and we might be able to help a little
 more.

 -chris

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org





-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Error in DBCP Connection Pool with tomcat 6.x

2014-07-02 Thread Phil Steitz
On 7/1/14, 10:48 PM, Vijendra Pachoriya wrote:
 Hi Filip,

 This error comes at some point of time on production, when the server is on 
 load.

 I am using hibernate and not closing any connection manually, its all managed 
 by hibernate and tomcat dbcp connection pool.

Two additional things to check:

1) Check logs to see if abandoned connections are being removed.  If
your code is closing all connections, you should not need this to be
enabled.
2) Make sure that there is no sharing across threads of the
Hibernate objects.

Phil

 For me it seems like DBCP is trying to close the connection which is already 
 closed by the database server.

 Please refer below related links.

 https://issues.apache.org/jira/browse/DBCP-329 

 http://markmail.org/message/c2gx2xuzum4pv743#query:+page:1+mid:ncvixx4ewe7pho2x+state:results
  


 Regards,
 Vijendra

 -Original Message-
 From: Filip Hanik [mailto:fi...@hanik.com] 
 Sent: 01 July 2014 21:06
 To: Tomcat Users List
 Subject: Re: Error in DBCP Connection Pool with tomcat 6.x

 Looks like your code already called java.sql.Connection.close() and then 
 attempts to use the connection again

 Filip



 On Tue, Jul 1, 2014 at 8:09 AM, Propes, Barry L barry.l.pro...@citi.com
 wrote:


 -Original Message-
 From: Vijendra Pachoriya [mailto:vijendra.pachor...@indegene.com]
 Sent: Tuesday, July 01, 2014 2:31 AM
 To: users@tomcat.apache.org
 Cc: Alok Roy
 Subject: Error in DBCP Connection Pool with tomcat 6.x

 Hi Tomcat Team,

 Please help me out in solving below error.

 Below is the details :

 Configuration in my 
 context.xml


 Resource name=jdbc/ABC
 auth=Container
 type=javax.sql.DataSource
 maxActive=50
 maxIdle=10
 maxWait=1
 username=ABC
 password=ABC
 removeAbandoned=true
 logAbandoned=true
 testOnBorrow=true
 testWhileIdle=true
 timeBetweenEvictionRunsMillis=3
 validationQuery=SELECT 1 FROM dual
 driverClassName=oracle.jdbc.driver.OracleDriver
 url=jdbc:oracle:thin:@MY_DB /


 ==Error Message 
 

 at
 org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
 at
 org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
 at
 org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
 at java.lang.Thread.run(Thread.java:679)
 Caused by: org.springframework.transaction.TransactionSystemException:
 Could not roll back JPA transaction; nested exception is
 javax.persistence.PersistenceException: unexpected error when rollbacking
 at
 org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:486)
 at
 org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:800)
 at
 org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:777)
 at
 org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:339)
 at
 org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
 at
 org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
 at
 org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
 at sun.proxy.$Proxy185.selectPharmaUser(Unknown Source)
 at
 com.aptilon.axcelrx.server.service.impl.AuthenticationServiceImpl.login(AuthenticationServiceImpl.java:170)
 at
 com.aptilon.axcelrx.server.ws.endpoint.AuthenticationEndpoint.login_aroundBody0(AuthenticationEndpoint.java:110)
 ... 45 more
 Caused by: javax.persistence.PersistenceException: unexpected error 
 when rollbacking
 at
 org.hibernate.ejb.TransactionImpl.rollback(TransactionImpl.java:88)
 at
 org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:482)
 ... 54 more
 Caused by: org.hibernate.exception.GenericJDBCException: Cannot 
 release connection
 at
 org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126)
 at
 org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114)
 at
 org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
 at
 org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
 at
 org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:478)
 at
 org.hibernate.jdbc.ConnectionManager.aggressiveRelease(ConnectionManager.java:429)
 at
 

Re: Not supported by Basic Datasource

2010-12-14 Thread Phil Steitz
On Tue, Dec 14, 2010 at 6:42 AM, SOPANMISHRA sopan.mis...@gmail.com wrote:


 1. I have posted the exact tags that i'm using in my hibernate.cfg.xml
 2. regarding the BasicDataSource.getConnection in DBCP-1.2.2 ,I have not
 mentioned username,password tag in my hibernate.cfg.xml so is there any
 other bug?


I am not a hibernate expert, but the DBCP exception that you are getting is
the result of the hibernate code calling getConnection(username,password),
as opposed to getConnectio().  The former is not supported by DBCP.  The
pool creates connections using credentials provided in its configuration.
These credentials, and the other JDBC connection properties, apply to all
connections managed by the pool.  You can't override them on individual
getConnection requests and they must be provided in the conifg.

Phil

3. Till now I have configured my files as asked by Chris  Mark
 4. I'm configuring application specific requirements in the
 D:\apache-tomcat-6.0.26\conf\context.xml
 5. If the data source works out then I will configure META-INF\context.xml
 for being project specific
 6. I tried removing property name=connection.pool_size10/property
 7. The error still prevails-Not supported by BasicDataSource

 --
 View this message in context:
 http://old.nabble.com/Not-supported-by-Basic-Datasource-tp30418811p30453889.html
 Sent from the Tomcat - User mailing list archive at Nabble.com.


 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org




Re: DBCP connections not in accordance with netstat output

2010-10-20 Thread Phil Steitz

On 10/20/10 5:32 AM, Felix Schumacher wrote:

Hi Marc,

Am Mittwoch, den 20.10.2010, 00:54 +0200 schrieb Marc Wilmots:

Hi List,

I installed Lambda Probe to debug a problem that I'm having with a Liferay
portal (5.1.2):

Tomcat: 6.0.26 with dbcp
JDK: 1.6.0_18
DB: Oracle 10.2.0.4 (ojdbc14)
RHEL 5.4 64Bits

When launching a stress test with JMeter, Lambda Probe indicates that my
connection pool (150 connections (I know..extremely high)) is out of free
connections (150 Busy and Established). However, when I do a netstat, there
are only 20-30 established connections to the database.

I would rather believe netstat over Probe, however when Lambda Probe
indicates that my pool is out of connections I noticed that the stress test
is going extremely slow and several requests are failing because of a
response time out (60secs). (The Tomcat connection pool (500 threads) gets
filled as well)

Try generating thread-dumps when your tomcat is under that load. That
way you can see where you might have problems.



Why could it be that Probe indicates 150 established connections and netstat
only 20-30?

I think Probe differentiates between established (pooled) connections
and busy (currently in use) connections. Maybe the established
connections are lazy and have no direct connection to the database.
Haven't checked it, so they very well be in another state, like
abandoned and cut down by a firewall... Your best bet is to look at the
thread-dumps.


According to the DBA, the DB is not stressed at all.

As soon as the stress test stops, the pool recovers itself. Netstat and
Lambda numbers (idle connections) are now exactly the same.

dbcp will shrink the pool to the desired minIdle settings when no new
connections are needed.


You mean maxIdle.  If abandoned connection cleanup is enabled, 
that will also eventually run at configured intervals, closing 
abandoned connections if they are still physically open and updating 
pool counters to reflect recovery.


Can you post your full DBCP configuration?  Have you verified that 
your code is closing all connections that it borrows from the pool?


Phil


HTH
  Felix


I know it's extremely (if not impossible) to indicate me what the problem
might be. I just would like to know how I should proceed to investigate this
problem, as I'm currently out of ideas.


Thanks in advance!




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: tomcat 6.0.29 hung

2010-10-20 Thread Phil Steitz

On 10/20/10 7:11 PM, Jason Britton wrote:

Thankfully when I pulled up jvisualvm on the server and issued thread dumps,
even though the stacktraces for the threads did not come up within jvisualvm
the thread stacktraces were dumped to stdout.  So I did get thread
stacktraces in my tomcat log.I won't post entire stacktraces here but
almost all blocked threads seem to be stuck blocking here:

 - waiting to lock0x2aaaced306f8  (a
org.apache.commons.dbcp.datasources.SharedPoolDataSource)

One of my threads in runnable state has that locked

 - locked0x2aaaced306f8  (a
org.apache.commons.dbcp.datasources.SharedPoolDataSource)


Looks like you are using Commons DBCP directly.  What versions of 
Commons DBCP and Commons Pool are you running?


Phil


So I'm pretty sure I've found the thread hanging everything up now to
take a look at the code and figure out why lock is not being released.

Highly recommend trying out jvisualvm (it's in your jdk bin) if you're
running 1.6.




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: DelegatingCallableStatement.getInnermostDelegate() -- NoSuchMethodError

2008-03-15 Thread Phil Steitz
On Fri, Mar 14, 2008 at 5:03 PM, Jim Garrison [EMAIL PROTECTED] wrote:

  I'm getting a NoSuchMethodError on
  org.apache.commons.dbcp.DelegatingCallableStatement.getInnermostDelegate
  ().

  AFAICT DelegatingCallableStatement inherits ultimately from
  DelegatingStatement, which DOES have such a method. I've examined the
  commons-dbcp-1.2.1.jar file that's in the classpath, and the
  inheritance hierarchy looks OK and there IS a getInnermostDelegate()
  method on DelegatingStatement.


  Since this is running in BusinessObjects' Tomcat 5.0 server I thought
  there might be a classloader problem.  I printed out the classloaders
  at the point where the error happens, but the debugging output seems
  to indicate that everything should work -- but I still get the
  exception.  I can't really post an SSCCE as this is buried in a much
  larger system that runs only under Tomcat.

  Source Fragment
  ===

  import java.sql.*;
  import org.apache.commons.dbcp.DelegatingCallableStatement;
 .
 .
 .
  PreparedStatement stmt;
 .
 .
 .
  protected void executeStatement() throws SQLException
  {
   if ( TPMonitor.TYPE_ORACLE == TPMonitor.getInstance().getDBType() )
   {
 System.out.println(DBTYPE: ORA);
 System.out.println(stmt CLASS:  + stmt.getClass().getName());
 System.out.println(stmt LOADER:  +
  stmt.getClass().getClassLoader().toString());

 DelegatingCallableStatement dcs = (DelegatingCallableStatement)
  stmt;
 System.out.println(dcs CLASS:  + dcs.getClass().getName());
 System.out.println(dcs LOADER:  +
  dcs.getClass().getClassLoader().toString());

 Statement cs = dcs.getInnermostDelegate();


  The above statement throws the following exception:

  java.lang.NoSuchMethodError:
  org.apache.commons.dbcp.DelegatingCallableStatement.getInnermostDelegate
  ()Ljava/sql/CallableStatement;



  Debugging Output
  

  DBTYPE: ORA
  stmt CLASS: org.apache.commons.dbcp.DelegatingCallableStatement
  stmt LOADER: StandardClassLoader
   delegate: true
   repositories:
 file:C:\Program Files\Business Objects\Tomcat\common\classes\
 file:C:\Program Files\Business
  Objects\Tomcat\common\endorsed\xalan.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\endorsed\xercesImpl.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\endorsed\xml-apis.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\ant-launcher.jar
 file:C:\Program Files\Business Objects\Tomcat\common\lib\ant.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\commons-collections-2.1.1.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\commons-dbcp-1.2.1.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\commons-el.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\commons-pool-1.2.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\jasper-compiler.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\jasper-runtime.jar
 file:C:\Program Files\Business Objects\Tomcat\common\lib\jsp-api.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\naming-common.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\naming-factory.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\naming-java.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\naming-resources.jar
 file:C:\Program Files\Business Objects\Tomcat\common\lib\ojdbc14.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\servlet-api.jar
 file:C:\Program Files\Business Objects\Tomcat\common\lib\sqljdbc.jar
  -- Parent Classloader:
  [EMAIL PROTECTED]

  dcs CLASS: org.apache.commons.dbcp.DelegatingCallableStatement
  dcs LOADER: StandardClassLoader
   delegate: true
   repositories:
 file:C:\Program Files\Business Objects\Tomcat\common\classes\
 file:C:\Program Files\Business
  Objects\Tomcat\common\endorsed\xalan.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\endorsed\xercesImpl.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\endorsed\xml-apis.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\ant-launcher.jar
 file:C:\Program Files\Business Objects\Tomcat\common\lib\ant.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\commons-collections-2.1.1.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\commons-dbcp-1.2.1.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\commons-el.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\commons-pool-1.2.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\jasper-compiler.jar
 file:C:\Program Files\Business
  Objects\Tomcat\common\lib\jasper-runtime.jar
 file:C:\Program Files\Business Objects\Tomcat\common\lib\jsp-api.jar
 file:C:\Program Files\Business
  

Re: connection pool maxActive = 0 != unlimited in new version

2008-03-08 Thread Phil Steitz
On Fri, Mar 7, 2008 at 3:41 PM, william kinney [EMAIL PROTECTED] wrote:
 Hi,

  After some painful research and code digging, I found that in commons-pool
  versions  1.2 (ie 1.3 and 1.4), the logic for maxActive changed for
  unlimited. 0 no longer means unlimited as it did in 1.2 and previous
  versions.

Correct.  This was changed in commons pool 1.3 to address JIRA issues
DBCP-113, POOL-22.

  This looks to be contradictory to commons-pool and tomcat documentation
  however (e.g.
  http://tomcat.apache.org/tomcat-5.5-doc/jndi-datasource-examples-howto.html
  ).
  This seems to effect tomcat versions 5.5.24-26. Are there plans to update
  said documentation, or is this an issue with commons-pool?


Thanks for reporting this.  The documentation update to reflect this
change for DBCP 1.2.2 was incorrect.  This issue is being tracked as
DBCP-41 (reopened).  The documentation fix will be applied and web
site updated shortly.

Phil

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Getting the URL of a DataSource

2008-03-04 Thread Phil Steitz
On Mon, Mar 3, 2008 at 2:08 PM, Katilie, John
[EMAIL PROTECTED] wrote:
 All, this may be a dumb question but I've exhausted my research and just
  wondering what other users out there are doing.

  I would like to get the URL for a DataSource to display for debugging
  and/or
  Informational reasons. I know I can get the URL after I get the
  DataSource and get a connection from the connection metadata. My problem
  is what if the
  Connection fails? It seems without the connection I can not get the URL
  for the DataSource.

  This is an application running under Tomcat 6.0.14. I guess I could get
  the URL from the JMX DataSource objects created by Tomcat (I can see
  them with jconsole). But is there another way? Am I missing something? I
  hope this question makes sense...

I may get flamed for condoning hacking here, but there is a
too-fragile-to-depend-on way to do this for debugging / testing.

If the DataSource implementation that you are using exposes a getUrl
method, and you really want to get this (or any other extended
property) in application code, you can get the URL by specializing the
cast on your JNDI lookup and then invoking that method.  For example,
if you are using the tomcat-provided DBCP DataSource,
org.apache.tomcat.dbcp.dbcp.BasicDataSource ds  =
(org.apache.tomcat.dbcp.dbcp.BasicDataSource)
initContext.lookup(java:/comp/env/jdbc/yourDB);
String urlString = ds.getUrl();
That obiously only works if you are using BasicDataSource bundled with
Tomcat or if whatever datasource you use supports something like this
(If you are using Commons DBCP directly, just change tomcat.dbcp to
commons).  It is not something you want to depend on, though, as all
Tomcat or your DS provider really promises you is a
javax.sql.DataSource.  In particular, there is no guarantee that code
such as the above will not start generating ClassCastExceptions or
ClassNotFound when the container (Tomcat) or datasource provider
release new versions.

Phil


  Thanks for any and all comments.

  Regards, John.

  -
  To start a new topic, e-mail: users@tomcat.apache.org
  To unsubscribe, e-mail: [EMAIL PROTECTED]
  For additional commands, e-mail: [EMAIL PROTECTED]



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: DBCP: Threads sitting forever in getConnection()?

2008-01-27 Thread Phil Steitz
On Jan 27, 2008 4:32 PM, Clay Collier [EMAIL PROTECTED] wrote:
 Hi all-

 I'm running into a recurring problem with an application that uses
 DBCP pooling to access a database.  The application initially seems to
 run fine; in the test environment, it has no problem dealing with a
 large volume of connections and responds to everything the way that
 you would expect.  However, on my production server exposed to users,
 the application runs fine for an indeterminate period of time, and
 then begins locking up.  Users send messages to the server, but never
 get any response back- not even an exception.  The manager's server
 status show lots of long-running threads sitting around for long
 periods of time.  Load on the Tomcat server and the database server
 are not a factor.  Looking at the thread dump seems to give this as
 the relevant bits (this from a dump with two stuck threads):

 http-8080-Processor25 daemon prio=10 tid=0x08a1c800 nid=0x4cd5 in
 Object.wait() [0xb4d3b000..0xb4d3c030]
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 0x8cf5c6c8 (a
 org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
 at java.lang.Object.wait(Object.java:485)
 at 
 org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810)
 - locked 0x8cf5c6c8 (a
 org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
 at 
 org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
 at 
 org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
 at 
 edu.stanford.hpn.servlet.NetPlumberServlet.doPost(NetPlumberServlet.java:76)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
 at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
 at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
 at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
 at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
 at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
 at 
 org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
 at 
 org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
 at 
 org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
 at 
 org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
 at 
 org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
 at java.lang.Thread.run(Thread.java:619)

 http-8080-Processor24 daemon prio=10 tid=0x08a1b800 nid=0x4cd4 in
 Object.wait() [0xb4d8c000..0xb4d8d0b0]
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 0x8cf5c6c8 (a
 org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
 at java.lang.Object.wait(Object.java:485)
 at 
 org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810)
 - locked 0x8cf5c6c8 (a
 org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
 at 
 org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
 at 
 org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
 at 
 edu.stanford.hpn.servlet.TopologyActivator.performAction(TopologyActivator.java:85)
 at 
 edu.stanford.hpn.servlet.NetPlumberServlet.doPost(NetPlumberServlet.java:99)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
 at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
 at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
 at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
 at