Re: Tips on identifying the DB connection leaks leading to the "Pool empty" error
Torsten, On 4/17/23 19:31, Torsten Krah wrote: Use logAbandoned as a boolean parameter +1 remoteAbandoned is only a band-aid; you need to fix your application. In development, I always run with maxTotal="1" and logAbandoned="true" and maxWaitMillis="1". This will help you find connection leaks very quickly. It will also help find connection deadlocks, where a single thread tries to obtain two connections from the pool instead of using only one at a time. This can cause a deadlock in the following situation: 0. Thread pool contains 2 connections 1. Thread 1 obtains a connection 2. Thread 2 obtains a connection 3. Thread 1 tries to obtain a connection and waits for it 4. Thread 2 tries to obtain a connection and waits for it Both threads are now waiting on the other thread to release one of the two available connections, and neither one ever will. Meanwhile, the pool is empty and nobody else can get any work done, either. I haven't posted this in a while, so: https://blog.christopherschultz.net/2009/03/16/properly-handling-pooled-jdbc-connections/ -chris Akshay Mishra schrieb am Di., 18. Apr. 2023, 01:07: Hello Tomcat Community! Tomcat Version: 9.0.70 Our Tomcat web service started exhausting the max connections allowed to the backend datastore (Postgres). From the logs, we don’t see any 1 thread requesting lot of connections. Nor do we see too many threads requesting a new connection. We collected thread dumps (at a fixed interval of 5 mins) and they do not point to too many threads requesting connections. We suspect someone could be leaking connections but we are unclear on how to narrow that down especially since the thread dump was inconclusive We added parameters like “removeAbandoned”, “removeAbandonedTimeout” and “ResetAbandonedTimer” to our datasource config and the results were encouraging - we don't see the pool getting exhausted but are concerned this might mask a larger problem in case there are connection leaks in our service. It would be great if the community has any thoughts on these new parameters. Or on way to narrow down the problem to isolate connection leaks. Any help is appreciated. Thanks Akshay - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tips on identifying the DB connection leaks leading to the "Pool empty" error
We stumbled into it to be honest. There should be a fairly easy way though
if you're on a linux box... something like..
grep -r ".getConnection" * should get you all the classes that make a call
to something like sqlConn.getConnection(), and then once you have that you
can check for finally {} blocks and ensure you;re closing out
Kevin Huntly
Email: kmhun...@gmail.com
Cell: 716/424-3311
-BEGIN GEEK CODE BLOCK-
Version: 1.0
GCS/IT d+ s a C++ UL+++$ P+(++) L+++ E---
W+++ N+ o K(+) w--- O- M-- V-- PS+ PE Y(+)
PGP++(+++) t+ 5-- X-- R+ tv+ b++ DI++ D++
G++ e(+) h--- r+++ y+++*
--END GEEK CODE BLOCK--
On Mon, Apr 17, 2023 at 7:32 PM Torsten Krah wrote:
> Use logAbandoned as a boolean parameter, you will get stacktraces in the
> log. From those you can narrow down where you are leaking connections.
>
> Akshay Mishra schrieb am Di., 18. Apr. 2023,
> 01:07:
>
> > Hello Tomcat Community!
> >
> >
> > Tomcat Version: 9.0.70
> >
> > Our Tomcat web service started exhausting the max connections allowed to
> > the backend datastore (Postgres).
> >
> > From the logs, we don’t see any 1 thread requesting lot of connections.
> > Nor do we see too many threads requesting a new connection.
> > We collected thread dumps (at a fixed interval of 5 mins) and they do not
> > point to too many threads requesting connections.
> >
> > We suspect someone could be leaking connections but we are unclear on how
> > to narrow that down especially since the thread dump was inconclusive
> >
> > We added parameters like “removeAbandoned”, “removeAbandonedTimeout” and
> > “ResetAbandonedTimer” to our datasource config and the results were
> > encouraging - we don't see the pool getting exhausted but are concerned
> > this might mask a larger problem in case there are connection leaks in
> our
> > service.
> >
> > It would be great if the community has any thoughts on these new
> > parameters. Or on way to narrow down the problem to isolate connection
> > leaks.
> >
> > Any help is appreciated. Thanks
> >
> > Akshay
> >
>
Re: Tips on identifying the DB connection leaks leading to the "Pool empty" error
Use logAbandoned as a boolean parameter, you will get stacktraces in the log. From those you can narrow down where you are leaking connections. Akshay Mishra schrieb am Di., 18. Apr. 2023, 01:07: > Hello Tomcat Community! > > > Tomcat Version: 9.0.70 > > Our Tomcat web service started exhausting the max connections allowed to > the backend datastore (Postgres). > > From the logs, we don’t see any 1 thread requesting lot of connections. > Nor do we see too many threads requesting a new connection. > We collected thread dumps (at a fixed interval of 5 mins) and they do not > point to too many threads requesting connections. > > We suspect someone could be leaking connections but we are unclear on how > to narrow that down especially since the thread dump was inconclusive > > We added parameters like “removeAbandoned”, “removeAbandonedTimeout” and > “ResetAbandonedTimer” to our datasource config and the results were > encouraging - we don't see the pool getting exhausted but are concerned > this might mask a larger problem in case there are connection leaks in our > service. > > It would be great if the community has any thoughts on these new > parameters. Or on way to narrow down the problem to isolate connection > leaks. > > Any help is appreciated. Thanks > > Akshay >
Re: Tips on identifying the DB connection leaks leading to the "Pool empty" error
my team saw this same thing in production - there was code that never closed the SQL connection after it was created... 150+ connections spike all at once, took out the pool. Not sure if that's your issue, but thought i'd mention it Kevin Huntly Email: kmhun...@gmail.com Cell: 716/424-3311 -BEGIN GEEK CODE BLOCK- Version: 1.0 GCS/IT d+ s a C++ UL+++$ P+(++) L+++ E--- W+++ N+ o K(+) w--- O- M-- V-- PS+ PE Y(+) PGP++(+++) t+ 5-- X-- R+ tv+ b++ DI++ D++ G++ e(+) h--- r+++ y+++* --END GEEK CODE BLOCK-- On Mon, Apr 17, 2023 at 7:07 PM Akshay Mishra wrote: > Hello Tomcat Community! > > > Tomcat Version: 9.0.70 > > Our Tomcat web service started exhausting the max connections allowed to > the backend datastore (Postgres). > > From the logs, we don’t see any 1 thread requesting lot of connections. > Nor do we see too many threads requesting a new connection. > We collected thread dumps (at a fixed interval of 5 mins) and they do not > point to too many threads requesting connections. > > We suspect someone could be leaking connections but we are unclear on how > to narrow that down especially since the thread dump was inconclusive > > We added parameters like “removeAbandoned”, “removeAbandonedTimeout” and > “ResetAbandonedTimer” to our datasource config and the results were > encouraging - we don't see the pool getting exhausted but are concerned > this might mask a larger problem in case there are connection leaks in our > service. > > It would be great if the community has any thoughts on these new > parameters. Or on way to narrow down the problem to isolate connection > leaks. > > Any help is appreciated. Thanks > > Akshay >
Tips on identifying the DB connection leaks leading to the "Pool empty" error
Hello Tomcat Community! Tomcat Version: 9.0.70 Our Tomcat web service started exhausting the max connections allowed to the backend datastore (Postgres). >From the logs, we don’t see any 1 thread requesting lot of connections. Nor do >we see too many threads requesting a new connection. We collected thread dumps (at a fixed interval of 5 mins) and they do not point to too many threads requesting connections. We suspect someone could be leaking connections but we are unclear on how to narrow that down especially since the thread dump was inconclusive We added parameters like “removeAbandoned”, “removeAbandonedTimeout” and “ResetAbandonedTimer” to our datasource config and the results were encouraging - we don't see the pool getting exhausted but are concerned this might mask a larger problem in case there are connection leaks in our service. It would be great if the community has any thoughts on these new parameters. Or on way to narrow down the problem to isolate connection leaks. Any help is appreciated. Thanks Akshay
