Am 24. Januar 2020 06:22:49 MEZ schrieb Jerry Malcolm <techst...@malcolms.com>:
>
>On 1/23/2020 11:04 PM, Jerry Malcolm wrote:
>> Sorry... forgot the version. You are correct. It's 8.5.x running on
>
>> AWS Linux EC2. I didn't intentionally configure anything special for
>
>> the pool. So I assume it's dbcp 2.0. Here is my resource config:
>>
>> <GlobalNamingResources>
>> <Resource name="jdbc/xxxxxxxx.com"
>>
>url="jdbc:mysql://xxxxxx-db-1.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com/xxxxxxxx"
>
>>
>> maxTotal="125"
>> maxIdle="3"
>> username="xxxxxx"
>> password="xxxxxxxxx"
>> auth="Container"
>> type="javax.sql.DataSource"
>> maxWaitMillis="30000"
>> removeAbandonedTimeout="5"
>> logAbandoned="true"
>> driverClassName="com.mysql.jdbc.Driver" />
>> </GlobalNamingResources>
The documentation for 8.5 mentions the parameters removeAbandonedOnBorrow and
removeAbandonedOnMaintenance. You need to set one of them to true.
Default for both is false, that is why you get your jmx values you showed us
below.
Regards
Felix
>>
>> I didn't notice the problem for a couple of days. Then it blew up on
>
>> a completely separate EC2 instance. It's a similar config, but only
>> one virtual host, and it's in development, not a production server,
>> so only a couple of developers are using it. I now have a log entry
>> from JMX that monitors numActive each time a connection is
>requested.
>> There have only been a handful of sessions over the past few days on
>> this server. For a couple of days, the log showed between 1 and 10
>> connections and always went back to 3 when idle. Then two days ago,
>in
>> one session, it went from 3 to 80. Then the server was idle for two
>> hours. When the next session started, it was still at 80 and went to
>
>> 110. Then 14 hours later there was another session that started with
>
>> 110 and went to 125 which resulted in a bunch of 'pool exhausted'
>> timeouts. It remained at 125 until I rebooted TC. I checked my RDS
>> database connections, and it's fluctuating around 10-20 connections.
>
>> So it appears the actual connections to the db are being closed. But
>
>> apparently when the trigger occurs that is causing this, pool
>> connections are being held forever. I realize that this is something
>
>> I could be doing. But I have one common entrance/exit module
>included
>> in every jsp that requests a connection on entry and returns it on
>> exit. If I have a leak in my code, it seems it would be consistently
>
>> recurring. And even if I never return any connection, shouldn't the
>> configuration above still clean up the abandoned connections after a
>> minute or so of idle?
>>
>> Is there any specific logging I can turn on that might show more of
>> what is going on in the pool?
>>
>> Thx
>>
>> Jerry
>>
>Hmm, I just expanded my jmx logging and I'm getting:
>
>AbandonedConfig : false
>LogAbandoned : false
>RemoveAbandonedOnBorrow : false
>RemoveAbandonedOnMaintenance : false
>RemoveAbandonedTimeout : 2147483647
>
>which obviously means I've got something big-time wrong in my
>configuration. It just occurred to me that I was running TC 9.x on my
>old Windows server and on my dev laptop, but AWS only would install
>8.5.x. So I completely forgot about the fact that a bunch of parameter
>
>names changed sometime back on resource definitions. I don't remember
>doing parameter translations when I moved to 8.5. I haven't looked it
>up yet. But do I have something stupid like a bunch of 9.x parameters
>that should be changed back to 8.5.x on my config? If the parameters
>are correct, then there's something else wrong to give me the jmx
>values
>above. But the parameters above definitely would give me the results
>I'm seeing.
>
>Am I on the right track?
>
>>
>> On 1/21/2020 12:49 PM, Christopher Schultz wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> Mark and Jerry,
>>>
>>> On 1/20/20 5:50 AM, Mark Thomas wrote:
>>>> Can you share the configuration of the connection pool? Don't
>>>> forget to mask any passwords.
>>> Also the Tomcat version ;)
>>>
>>> Recent posts from you suggest that this is 8.5.x, which means you
>are
>>> using commons-dbcp 2.0 unless you have specifically configured
>>> tomcat-pool (which will be obvious from your configuration).
>>>
>>> But it's always good to be sure.
>>>
>>> - -chris
>>>
>>>> On 19/01/2020 05:43, Jerry Malcolm wrote:
>>>>> I have a web page that makes a couple of hundred ajax calls when
>>>>> it loads. But the calls are recursive. The response from one
>>>>> call generates a couple of more calls. The responses from those
>>>>> calls generate others, etc. So this is not a blast of 200
>>>>> simultaneous calls to the server. In most cases the count of
>>>>> active database connections never gets over 10-15 at a time. I
>>>>> have the max count on the connection pool set to 125.
>>>>>
>>>>> The server has very low traffic averaging a few pages an hour.
>>>>> Yesterday, I was testing this one page described above and and I
>>>>> started getting errors that no connections were available in the
>>>>> pool. My connection abort timeouts are very short, about 5
>>>>> seconds, and the wait time for a free connection is 60 seconds, I
>>>>> believe. So theoretically, even if there were connection leaks,
>>>>> they should be returned well before waiting connections timed
>>>>> out, right? I went to the AWS RDS console to look at the
>>>>> connections graph. It showed that in the previous couple of
>>>>> minutes the database went from 3 connections to the max of 125.
>>>>> The page had timeout errors. But it was done. There was no more
>>>>> activity on this webapp. Yet the database monitor continued to
>>>>> show 125 connections. I kept refreshing for about 10 minutes
>>>>> expecting the connection pool to finally drop all of the
>>>>> connections. Nothing. Finally after about 10 minutes it still
>>>>> showed 125 connections, I bounced tomcat and the connection count
>>>>> on the database immediately fell to zero. BTW, I have
>>>>> logAbandoned set to true, and I'm not getting any abandon log
>>>>> entries.
>>>>>
>>>>> The problem occurred twice requiring a reboot of TC each time.
>>>>> I enabled some jmx code in my web app that logs the connection
>>>>> count each time I request/return a connection. Of course, now
>>>>> that I've got logging on, I can't reproduce the problem. I can
>>>>> reload that page a hundred times now, and my AWS RDS monitor
>>>>> shows a bump up to 10-15 connections for a brief time, then
>>>>> returns, exactly like I believe it should be working.
>>>>>
>>>>> I'm not a big fan of problems that just go away. I really want
>>>>> to figure out what is happening here, even if it's random. I
>>>>> don't believe this is a problem in the mainline code. Otherwise
>>>>> I think it would be more consistent. And again, even if I was
>>>>> not returning connections, they should be timing out and being
>>>>> logged. Another option is that I have a rogue thread that is
>>>>> sucking up all of the connections and holding them. But this
>>>>> problem only starts when I load this page. And I'm not aware of
>>>>> code that starts new threads in that area. Is there anything I
>>>>> can be doing to the connection pool to make it not close
>>>>> connections and return them to the pool? Or is there anything
>>>>> the database could be doing that would cause closing a connection
>>>>> to fail?
>>>>>
>>>>> I guess I'm just grasping at straws. Is there any type of
>>>>> low-level logging that I can enable that will tell me each time a
>>>>> connection is requested and returned (and possibly the call stack
>>>>> each time)?
>>>>>
>>>>> I've got a feeling this one is going to bite me some time soon
>>>>> and take a client site down with it. I really need to understand
>>>>> this one.
>>>>>
>>>>> Thanks as always,
>>>>>
>>>>> Jerry
>>>>>
>>>>>
>>>>>
>---------------------------------------------------------------------
>>>>>
>>>>>
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>>>
>>>>
>>>>
>---------------------------------------------------------------------
>>>>
>>>>
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>>
>>> -----BEGIN PGP SIGNATURE-----
>>> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>>>
>>> iQIyBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl4nR8gACgkQHPApP6U8
>>> pFjCvw/434FFGqDusIzEGtbo/DFwwSAzoi1mbkeZvZ1fui0A0/wU3CCLgvE9eBVc
>>> Y7GMIByTvNIEuDu3SpN+XLRdDvMcn+OJmlTnJmGiOMy2WZKr6lAL/U57YjUlpNgj
>>> BiTx2Y26ohbfmWBfy32x6Y3s5r4aszjyOMIoF3RNzgFa1YhvezNpyNxBqjoyxARi
>>> lU5djVbo8mdzZEORYifQdAZNqFviQLdRN+Dp6+j0+g8Y0xCIK9gJp3OtJxNL0Nbc
>>> FgGIIlO6V7uLgIXEPCnMkV4R5h6vrQ9feDjbxBhsafHfbqzHpCoWvWZyrYcyxC51
>>> CimYxyenWUabh0TpGNxUovMgqEbdOrXQzQF/ALWBwyKtJevj4J/MuUNBFLluyt9K
>>> dW2CUtZrG+6eo0Ns65PAZ9jSPQNOJX0D8HYbCQj1QpRn6LMtfPmAMhveOjiZSDBu
>>> BXzRMPK9yqsJoJiziNNiFV2qF1h+MZQDwiIbUttWnm1bfnWGrBix/+wC+Kkr2rR/
>>> ChzQFebHRdoxJGwqGZ8iaXBqlNYYIpp6OssCuRFEsDLBuPuDtupr48IstGsZm9Ni
>>> ZuRTEhD2Agv0yXS0uTyS4OK2cFuoOGCZ59QB7553dQngXDtCoGbdQ8Fi5iJ4oCpB
>>> 4jF9vTnOzOjtpqacN98EpDPMHOQ9H+RDNzluVL5haD0DTybmdg==
>>> =/J/H
>>> -----END PGP SIGNATURE-----
>>>
>>>
>---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
