Re: Datasource Connections Problem
Am 24. Januar 2020 06:22:49 MEZ schrieb Jerry Malcolm : > >On 1/23/2020 11:04 PM, Jerry Malcolm wrote: >> Sorry... forgot the version. You are correct. It's 8.5.x running on > >> AWS Linux EC2. I didn't intentionally configure anything special for > >> the pool. So I assume it's dbcp 2.0. Here is my resource config: >> >> >> > >url="jdbc:mysql://xx-db-1..us-east-1.rds.amazonaws.com/" > >> >> maxTotal="125" >> maxIdle="3" >> username="xx" >> password="x" >> auth="Container" >> type="javax.sql.DataSource" >> maxWaitMillis="3" >> removeAbandonedTimeout="5" >> logAbandoned="true" >> driverClassName="com.mysql.jdbc.Driver" /> >> The documentation for 8.5 mentions the parameters removeAbandonedOnBorrow and removeAbandonedOnMaintenance. You need to set one of them to true. Default for both is false, that is why you get your jmx values you showed us below. Regards Felix >> >> I didn't notice the problem for a couple of days. Then it blew up on > >> a completely separate EC2 instance. It's a similar config, but only >> one virtual host, and it's in development, not a production server, >> so only a couple of developers are using it. I now have a log entry >> from JMX that monitors numActive each time a connection is >requested. >> There have only been a handful of sessions over the past few days on >> this server. For a couple of days, the log showed between 1 and 10 >> connections and always went back to 3 when idle. Then two days ago, >in >> one session, it went from 3 to 80. Then the server was idle for two >> hours. When the next session started, it was still at 80 and went to > >> 110. Then 14 hours later there was another session that started with > >> 110 and went to 125 which resulted in a bunch of 'pool exhausted' >> timeouts. It remained at 125 until I rebooted TC. I checked my RDS >> database connections, and it's fluctuating around 10-20 connections. > >> So it appears the actual connections to the db are being closed. But > >> apparently when the trigger occurs that is causing this, pool >> connections are being held forever. I realize that this is something > >> I could be doing. But I have one common entrance/exit module >included >> in every jsp that requests a connection on entry and returns it on >> exit. If I have a leak in my code, it seems it would be consistently > >> recurring. And even if I never return any connection, shouldn't the >> configuration above still clean up the abandoned connections after a >> minute or so of idle? >> >> Is there any specific logging I can turn on that might show more of >> what is going on in the pool? >> >> Thx >> >> Jerry >> >Hmm, I just expanded my jmx logging and I'm getting: > >AbandonedConfig : false >LogAbandoned : false >RemoveAbandonedOnBorrow : false >RemoveAbandonedOnMaintenance : false >RemoveAbandonedTimeout : 2147483647 > >which obviously means I've got something big-time wrong in my >configuration. It just occurred to me that I was running TC 9.x on my >old Windows server and on my dev laptop, but AWS only would install >8.5.x. So I completely forgot about the fact that a bunch of parameter > >names changed sometime back on resource definitions. I don't remember >doing parameter translations when I moved to 8.5. I haven't looked it >up yet. But do I have something stupid like a bunch of 9.x parameters >that should be changed back to 8.5.x on my config? If the parameters >are correct, then there's something else wrong to give me the jmx >values >above. But the parameters above definitely would give me the results >I'm seeing. > >Am I on the right track? > >> >> On 1/21/2020 12:49 PM, Christopher Schultz wrote: >>> -BEGIN PGP SIGNED MESSAGE- >>> Hash: SHA256 >>> >>> Mark and Jerry, >>> >>> On 1/20/20 5:50 AM, Mark Thomas wrote: Can you share the configuration of the connection pool? Don't forget to mask any passwords. >>> Also the Tomcat version ;) >>> >>> Recent posts from you suggest that this is 8.5.x, which means you >are >>> using commons-dbcp 2.0 unless you have specifically configured >>> tomcat-pool (which will be obvious from your configuration). >>> >>> But it's always good to be sure. >>> >>> - -chris >>> On 19/01/2020 05:43, Jerry Malcolm wrote: > I have a web page that makes a couple of hundred ajax calls when > it loads. But the calls are recursive. The response from one > call generates a couple of more calls. The responses from those > calls generate others, etc. So this is not a blast of 200 > simultaneous calls to the server. In most cases the count of > active database connections never gets over 10-15 at a time. I > have the max count on the connection pool set to 125. > > The server has very
Re: Datasource Connections Problem
On 1/23/2020 11:04 PM, Jerry Malcolm wrote: Sorry... forgot the version. You are correct. It's 8.5.x running on AWS Linux EC2. I didn't intentionally configure anything special for the pool. So I assume it's dbcp 2.0. Here is my resource config: url="jdbc:mysql://xx-db-1..us-east-1.rds.amazonaws.com/" maxTotal="125" maxIdle="3" username="xx" password="x" auth="Container" type="javax.sql.DataSource" maxWaitMillis="3" removeAbandonedTimeout="5" logAbandoned="true" driverClassName="com.mysql.jdbc.Driver" /> I didn't notice the problem for a couple of days. Then it blew up on a completely separate EC2 instance. It's a similar config, but only one virtual host, and it's in development, not a production server, so only a couple of developers are using it. I now have a log entry from JMX that monitors numActive each time a connection is requested. There have only been a handful of sessions over the past few days on this server. For a couple of days, the log showed between 1 and 10 connections and always went back to 3 when idle. Then two days ago, in one session, it went from 3 to 80. Then the server was idle for two hours. When the next session started, it was still at 80 and went to 110. Then 14 hours later there was another session that started with 110 and went to 125 which resulted in a bunch of 'pool exhausted' timeouts. It remained at 125 until I rebooted TC. I checked my RDS database connections, and it's fluctuating around 10-20 connections. So it appears the actual connections to the db are being closed. But apparently when the trigger occurs that is causing this, pool connections are being held forever. I realize that this is something I could be doing. But I have one common entrance/exit module included in every jsp that requests a connection on entry and returns it on exit. If I have a leak in my code, it seems it would be consistently recurring. And even if I never return any connection, shouldn't the configuration above still clean up the abandoned connections after a minute or so of idle? Is there any specific logging I can turn on that might show more of what is going on in the pool? Thx Jerry Hmm, I just expanded my jmx logging and I'm getting: AbandonedConfig : false LogAbandoned : false RemoveAbandonedOnBorrow : false RemoveAbandonedOnMaintenance : false RemoveAbandonedTimeout : 2147483647 which obviously means I've got something big-time wrong in my configuration. It just occurred to me that I was running TC 9.x on my old Windows server and on my dev laptop, but AWS only would install 8.5.x. So I completely forgot about the fact that a bunch of parameter names changed sometime back on resource definitions. I don't remember doing parameter translations when I moved to 8.5. I haven't looked it up yet. But do I have something stupid like a bunch of 9.x parameters that should be changed back to 8.5.x on my config? If the parameters are correct, then there's something else wrong to give me the jmx values above. But the parameters above definitely would give me the results I'm seeing. Am I on the right track? On 1/21/2020 12:49 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark and Jerry, On 1/20/20 5:50 AM, Mark Thomas wrote: Can you share the configuration of the connection pool? Don't forget to mask any passwords. Also the Tomcat version ;) Recent posts from you suggest that this is 8.5.x, which means you are using commons-dbcp 2.0 unless you have specifically configured tomcat-pool (which will be obvious from your configuration). But it's always good to be sure. - -chris On 19/01/2020 05:43, Jerry Malcolm wrote: I have a web page that makes a couple of hundred ajax calls when it loads. But the calls are recursive. The response from one call generates a couple of more calls. The responses from those calls generate others, etc. So this is not a blast of 200 simultaneous calls to the server. In most cases the count of active database connections never gets over 10-15 at a time. I have the max count on the connection pool set to 125. The server has very low traffic averaging a few pages an hour. Yesterday, I was testing this one page described above and and I started getting errors that no connections were available in the pool. My connection abort timeouts are very short, about 5 seconds, and the wait time for a free connection is 60 seconds, I believe. So theoretically, even if there were connection leaks, they should be returned well before waiting connections timed out, right? I went to the AWS RDS console to look at the connections graph. It showed that in the previous couple of minutes the database went from 3 connections to the max of 125. The
Re: Datasource Connections Problem
Sorry... forgot the version. You are correct. It's 8.5.x running on AWS Linux EC2. I didn't intentionally configure anything special for the pool. So I assume it's dbcp 2.0. Here is my resource config: I didn't notice the problem for a couple of days. Then it blew up on a completely separate EC2 instance. It's a similar config, but only one virtual host, and it's in development, not a production server, so only a couple of developers are using it. I now have a log entry from JMX that monitors numActive each time a connection is requested. There have only been a handful of sessions over the past few days on this server. For a couple of days, the log showed between 1 and 10 connections and always went back to 3 when idle. Then two days ago, in one session, it went from 3 to 80. Then the server was idle for two hours. When the next session started, it was still at 80 and went to 110. Then 14 hours later there was another session that started with 110 and went to 125 which resulted in a bunch of 'pool exhausted' timeouts. It remained at 125 until I rebooted TC. I checked my RDS database connections, and it's fluctuating around 10-20 connections. So it appears the actual connections to the db are being closed. But apparently when the trigger occurs that is causing this, pool connections are being held forever. I realize that this is something I could be doing. But I have one common entrance/exit module included in every jsp that requests a connection on entry and returns it on exit. If I have a leak in my code, it seems it would be consistently recurring. And even if I never return any connection, shouldn't the configuration above still clean up the abandoned connections after a minute or so of idle? Is there any specific logging I can turn on that might show more of what is going on in the pool? Thx Jerry On 1/21/2020 12:49 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark and Jerry, On 1/20/20 5:50 AM, Mark Thomas wrote: Can you share the configuration of the connection pool? Don't forget to mask any passwords. Also the Tomcat version ;) Recent posts from you suggest that this is 8.5.x, which means you are using commons-dbcp 2.0 unless you have specifically configured tomcat-pool (which will be obvious from your configuration). But it's always good to be sure. - -chris On 19/01/2020 05:43, Jerry Malcolm wrote: I have a web page that makes a couple of hundred ajax calls when it loads. But the calls are recursive. The response from one call generates a couple of more calls. The responses from those calls generate others, etc. So this is not a blast of 200 simultaneous calls to the server. In most cases the count of active database connections never gets over 10-15 at a time. I have the max count on the connection pool set to 125. The server has very low traffic averaging a few pages an hour. Yesterday, I was testing this one page described above and and I started getting errors that no connections were available in the pool. My connection abort timeouts are very short, about 5 seconds, and the wait time for a free connection is 60 seconds, I believe. So theoretically, even if there were connection leaks, they should be returned well before waiting connections timed out, right? I went to the AWS RDS console to look at the connections graph. It showed that in the previous couple of minutes the database went from 3 connections to the max of 125. The page had timeout errors. But it was done. There was no more activity on this webapp. Yet the database monitor continued to show 125 connections. I kept refreshing for about 10 minutes expecting the connection pool to finally drop all of the connections. Nothing. Finally after about 10 minutes it still showed 125 connections, I bounced tomcat and the connection count on the database immediately fell to zero. BTW, I have logAbandoned set to true, and I'm not getting any abandon log entries. The problem occurred twice requiring a reboot of TC each time. I enabled some jmx code in my web app that logs the connection count each time I request/return a connection. Of course, now that I've got logging on, I can't reproduce the problem. I can reload that page a hundred times now, and my AWS RDS monitor shows a bump up to 10-15 connections for a brief time, then returns, exactly like I believe it should be working. I'm not a big fan of problems that just go away. I really want to figure out what is happening here, even if it's random. I don't believe this is a problem in the mainline code. Otherwise I think it would be more consistent. And again, even if I was not returning connections, they should be timing out and being logged. Another option is that I have a rogue thread that is sucking up all of the connections and holding them. But this problem only starts when I load this page. And I'm not aware of code that starts new threads in
RE: Tomcat 9.0.27 intermittent JVM crashing on libapr-1.so
Hi Michael, Thank you I will update and observe, interesting that the crash only happens on this machine, we have 3 other machines with the same setup which I don't see the crashes. Warm regards, Norbert Elbanbuena -Original Message- From: Michael Osipov Sent: Thursday, January 23, 2020 1:52 AM To: Tomcat Users List Subject: Re: Tomcat 9.0.27 intermittent JVM crashing on libapr-1.so Am 2020-01-22 um 23:30 schrieb Norbert Elbanbuena: > Hello and greetings, > > I'm running tomcat 9.0.27 using APR and OpenSSL and get intermittent crashes > with the hs_err log always pointing that the problematic frame is libapr-1.so. > > Our machine details: > OS: CentOS Linux release 7.6.1810 (Core) > Java: OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) APR > based Apache Tomcat Native library [1.2.23] using APR version [1.4.8] > OpenSSL 1.0.2k-fips > > On startup, I don't see any errors on APR/OpenSSL > > 22-Jan-2020 20:29:14.188 INFO [main] > org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based > Apache Tomcat Native library [1.2.23] using APR version [1.4.8]. You should first upgrade your APR version to at least 1.6.x. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi, we started noticing that HttpServletRequest.getRemoteAddr() was sometimes returning NULL (which is invalid according to the servlet spec), about 20-50 times per day (we have high-load servers which routinely handle over 100 requests per second). This only happens on secure HTTP/2 requests. We noticed this while running the latest 9.0.30, but looking at our logs for the past few months, this was also happening on 9.0.21 as well as 9.0.22 which are the previous versions we had deployed. This issue was not present in 9.0.17, which was the version we were using before 9.0.21. Before anyone cries out "there is a bug in your code", Tomcat itself reports a NULL remoteAddr via AccessLogValve in these cases. It is also interesting that the remotePort is reported as "-1". We looked through the changelog, but we can't figure out what change might have triggered this (presumably some change between 9.0.17 and 9.0.21). It looks like some kind of async race condition during the HTTP/2 upgrade, but this is only an educated guess. Our server.xml is below (confidential data has been modified). We are using the NIO connector, Tomcat Native + APR libraries (but NOT the APR connector). The useAsyncIO flag is disabled because of server lockups we've experienced with this flag enabled on our production servers. Any ideas? - Manuel Dominguez Sarmiento
Re: Tomcat9, JSP, CSS and JS not loading in Firefox
On 22.01.2020 10:26, Lmhelp1 wrote: Hello Chris and all, Sorry for my late answer. Thank you for the link you suggested me to read. Adding the element: UTF-8 to "web.xml" solved my problem. Glad to hear that. From an absolute point of view, this is of course again a "patch". But as Chris pointed out before, it is unfortunately an unavoidable one, due to the fact that the browsers lamentably fail to always indicate the character set and encoding that they are using for sending text data to the server. [RANT:] (Although they know, and although the specifications provide them with an easy way to do that. Why that is so is anyone's guess, but the economic losses due to that over the years must be staggering). (probably close to the cost of allowing spaces in directories and files names) On 17/01/2020 3:17 PM, Christopher Schultz wrote: Your filter changes the encoding of both the request AND the response. It's likely that fixing the request-encoding was necessary, while changing the response-encoding was not. The problem is: 1. The official RFC-defined default character encoding for HTTP is ISO-8859-1. 2. The browser default appears to be unpredictable, but often UTF-8. 3. The browsers have all agreed not to tell the server what characted encoding is actually being used. It's fun. It's a very simply-solved problem: the browser should just advertise the character encoding and everything would be great. Sadly, n o. I would encourage you to read this page in its entirety: https://cwiki.apache.org/confluence/display/TOMCAT/Character+Encoding - -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
