Sorry... forgot the version. You are correct. It's 8.5.x running on
AWS Linux EC2. I didn't intentionally configure anything special for
the pool. So I assume it's dbcp 2.0. Here is my resource config:
<GlobalNamingResources>
<Resource name="jdbc/xxxxxxxx.com"
url="jdbc:mysql://xxxxxx-db-1.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com/xxxxxxxx"
maxTotal="125"
maxIdle="3"
username="xxxxxx"
password="xxxxxxxxx"
auth="Container"
type="javax.sql.DataSource"
maxWaitMillis="30000"
removeAbandonedTimeout="5"
logAbandoned="true"
driverClassName="com.mysql.jdbc.Driver" />
</GlobalNamingResources>
I didn't notice the problem for a couple of days. Then it blew up on a
completely separate EC2 instance. It's a similar config, but only one
virtual host, and it's in development, not a production server, so only
a couple of developers are using it. I now have a log entry from JMX
that monitors numActive each time a connection is requested. There have
only been a handful of sessions over the past few days on this server.
For a couple of days, the log showed between 1 and 10 connections and
always went back to 3 when idle. Then two days ago, in one session, it
went from 3 to 80. Then the server was idle for two hours. When the
next session started, it was still at 80 and went to 110. Then 14 hours
later there was another session that started with 110 and went to 125
which resulted in a bunch of 'pool exhausted' timeouts. It remained at
125 until I rebooted TC. I checked my RDS database connections, and
it's fluctuating around 10-20 connections. So it appears the actual
connections to the db are being closed. But apparently when the trigger
occurs that is causing this, pool connections are being held forever. I
realize that this is something I could be doing. But I have one common
entrance/exit module included in every jsp that requests a connection on
entry and returns it on exit. If I have a leak in my code, it seems it
would be consistently recurring. And even if I never return any
connection, shouldn't the configuration above still clean up the
abandoned connections after a minute or so of idle?
Is there any specific logging I can turn on that might show more of what
is going on in the pool?
Thx
Jerry
On 1/21/2020 12:49 PM, Christopher Schultz wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark and Jerry,
On 1/20/20 5:50 AM, Mark Thomas wrote:
Can you share the configuration of the connection pool? Don't
forget to mask any passwords.
Also the Tomcat version ;)
Recent posts from you suggest that this is 8.5.x, which means you are
using commons-dbcp 2.0 unless you have specifically configured
tomcat-pool (which will be obvious from your configuration).
But it's always good to be sure.
- -chris
On 19/01/2020 05:43, Jerry Malcolm wrote:
I have a web page that makes a couple of hundred ajax calls when
it loads. But the calls are recursive. The response from one
call generates a couple of more calls. The responses from those
calls generate others, etc. So this is not a blast of 200
simultaneous calls to the server. In most cases the count of
active database connections never gets over 10-15 at a time. I
have the max count on the connection pool set to 125.
The server has very low traffic averaging a few pages an hour.
Yesterday, I was testing this one page described above and and I
started getting errors that no connections were available in the
pool. My connection abort timeouts are very short, about 5
seconds, and the wait time for a free connection is 60 seconds, I
believe. So theoretically, even if there were connection leaks,
they should be returned well before waiting connections timed
out, right? I went to the AWS RDS console to look at the
connections graph. It showed that in the previous couple of
minutes the database went from 3 connections to the max of 125.
The page had timeout errors. But it was done. There was no more
activity on this webapp. Yet the database monitor continued to
show 125 connections. I kept refreshing for about 10 minutes
expecting the connection pool to finally drop all of the
connections. Nothing. Finally after about 10 minutes it still
showed 125 connections, I bounced tomcat and the connection count
on the database immediately fell to zero. BTW, I have
logAbandoned set to true, and I'm not getting any abandon log
entries.
The problem occurred twice requiring a reboot of TC each time.
I enabled some jmx code in my web app that logs the connection
count each time I request/return a connection. Of course, now
that I've got logging on, I can't reproduce the problem. I can
reload that page a hundred times now, and my AWS RDS monitor
shows a bump up to 10-15 connections for a brief time, then
returns, exactly like I believe it should be working.
I'm not a big fan of problems that just go away. I really want
to figure out what is happening here, even if it's random. I
don't believe this is a problem in the mainline code. Otherwise
I think it would be more consistent. And again, even if I was
not returning connections, they should be timing out and being
logged. Another option is that I have a rogue thread that is
sucking up all of the connections and holding them. But this
problem only starts when I load this page. And I'm not aware of
code that starts new threads in that area. Is there anything I
can be doing to the connection pool to make it not close
connections and return them to the pool? Or is there anything
the database could be doing that would cause closing a connection
to fail?
I guess I'm just grasping at straws. Is there any type of
low-level logging that I can enable that will tell me each time a
connection is requested and returned (and possibly the call stack
each time)?
I've got a feeling this one is going to bite me some time soon
and take a client site down with it. I really need to understand
this one.
Thanks as always,
Jerry
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIyBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl4nR8gACgkQHPApP6U8
pFjCvw/434FFGqDusIzEGtbo/DFwwSAzoi1mbkeZvZ1fui0A0/wU3CCLgvE9eBVc
Y7GMIByTvNIEuDu3SpN+XLRdDvMcn+OJmlTnJmGiOMy2WZKr6lAL/U57YjUlpNgj
BiTx2Y26ohbfmWBfy32x6Y3s5r4aszjyOMIoF3RNzgFa1YhvezNpyNxBqjoyxARi
lU5djVbo8mdzZEORYifQdAZNqFviQLdRN+Dp6+j0+g8Y0xCIK9gJp3OtJxNL0Nbc
FgGIIlO6V7uLgIXEPCnMkV4R5h6vrQ9feDjbxBhsafHfbqzHpCoWvWZyrYcyxC51
CimYxyenWUabh0TpGNxUovMgqEbdOrXQzQF/ALWBwyKtJevj4J/MuUNBFLluyt9K
dW2CUtZrG+6eo0Ns65PAZ9jSPQNOJX0D8HYbCQj1QpRn6LMtfPmAMhveOjiZSDBu
BXzRMPK9yqsJoJiziNNiFV2qF1h+MZQDwiIbUttWnm1bfnWGrBix/+wC+Kkr2rR/
ChzQFebHRdoxJGwqGZ8iaXBqlNYYIpp6OssCuRFEsDLBuPuDtupr48IstGsZm9Ni
ZuRTEhD2Agv0yXS0uTyS4OK2cFuoOGCZ59QB7553dQngXDtCoGbdQ8Fi5iJ4oCpB
4jF9vTnOzOjtpqacN98EpDPMHOQ9H+RDNzluVL5haD0DTybmdg==
=/J/H
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
