Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys
Hi Chris *This wasn't the case for httpd for many years. I don't know what itdoes these days, but it used to reply with a nice "400 Bad Request"error just like Tomcat is doing. The difference is that httpd has richconfiguration options to allow you to override that behavior. * Correct. By default HTTP also gives 400 Bad Request message. But we can override this behavior by using ErrorDocument directive to redirect to https URL. Currently in Tomcat this feature is not available. Tomcat has RewriteValve but it does not work in this case. On Fri, Aug 28, 2020 at 12:30 AM Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Merka, > > On 8/27/20 06:32, Phoenix, Merka wrote: > > I think what the Qualys scan is trying to flag is that the server > > (Tomcat) is listening for both secured and unsecured traffic on > > the _same_ TCP port when the server should be listening for just > > one of the two options (unsecured or secured), not both. > This actually might be a semi-legitimate test. If the client says "Our > policy is to only communicate using encrypted connections" and the > test says "well, here's a non-encrypted connection right here!" then > it makes some small amount of sense. In that case, it's all driven by > policy, and the policy can say "we have a carve-out for TLS handshake > failures" and then allow that particular test to pass. > > > The error message returned by the Tomcat service, while certainly > > helpful to the remote client, is returning more information than > > it should (from a security-viewpoint). > Not really. > > > If the default behavior for Tomcat is to return this "helpful" > > message for misconfigured clients, would it be reasonable for the > > Connector to have an option (attribute) for turning off this > > feature and simply reject with a TLS error any unsecured traffic on > > the port? This would address the security concern without imposing > > too much "bloat" to the Tomcat side. > I think this might actually be better/easier than implementing a > redirect. It's a simple flag that says "return nice errors on > plaintext-over-TLS connection attempts" (or similar) and the only > thing that changes is that we STOP trying to be nice to the client if > the setting is set to "fail" versus "be-nice". > > > For most other web servers (Apache httpd, NGINX, etc.) that offer > > https, the normal behavior is that when an http client tries to > > speak http to server expecting https, the client sees some garbled > > text (the server's TLS response to the connection attempt). > This wasn't the case for httpd for many years. I don't know what it > does these days, but it used to reply with a nice "400 Bad Request" > error just like Tomcat is doing. The difference is that httpd has rich > configuration options to allow you to override that behavior. > > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H7ZkACgkQHPApP6U8 > pFgOng/8DtMJkQc+MYLRm1iuCD7GZ2f2S7oQ59vFeCGAbmkiUjESvQ42QRGqIMy8 > 47giFc3ERm84DxyyyU7O/YDFxinOnCrC/v9A6RzpYKlZBSOq9Oy6732xTUAqGGIw > +3QGPXvyjE2Vcg1iavW+7cUKN1Q9R1NGcDRRBpBL0KRQMA3NV9pxGU71/In8TQvi > VZ51f7VTNXaJ2l+w6G23XBJkKQk3csFixevVzr4Xr56FLfqPxUc3m6QNu4BaHmgb > c95hceV/N7tkR9yHdaRtahpZq0lhGqXbNXfqjf7kElSkRmeAZ3MSsdnFD4fBHThn > xvz204xffSE71Z4W24W9gx23+Bg0y2EfPRo1CWC93rEvNRKMK6ILzLczTRA0w6QW > 9zP1XC+VwC25LQOGFDgFukQVupPYiMoNSb6DRey5ZUhur6v25nevwbhM0QsAm/oO > oZpreKaUMy+ZoixwGhaZ+UFiZRav7DRLSj85BjK9PqcP4VdPzFR9MarvMqLPxRoq > YxL/jNet4L+29Z2tDkZv4gfGJqI7oWkfXUsBFBjj5JgXrqE94Q8PzAK87pLeU80y > p3IL4krovHbu01j1fE3/aDotEvBu/wxWTCWze9+vL09a82PuTT2pihyCVqFuP9rS > kP0DtVTfbaUMyD2dryjyw4q1NdLYht4y/HHkyU/3cPopCbxEopU= > =4F4z > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
On Thu, Aug 27, 2020 at 4:30 PM Christopher Schultz wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > David, > > On 8/27/20 17:14, David wrote: > > Thank you all for the replies! > > > > On Thu, Aug 27, 2020 at 3:53 PM Christopher Schultz > > wrote: > >> > > David, > > > > On 8/27/20 13:57, David wrote: > On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz > wrote: > > > David, > > On 8/27/20 10:48, David wrote: > >>> In the last two weeks I've had two occurrences where a > >>> single CentOS 7 production server hosting a public > >>> webpage has become unresponsive. The first time, all > >>> 300 available "https-jsse-nio-8443" threads were > >>> consumed, with the max age being around 45minutes, and > >>> all in a "S" status. This time all 300 were consumed in > >>> "S" status with the oldest being around ~16minutes. A > >>> restart of Tomcat on both occasions freed these threads > >>> and the website became responsive again. The > >>> connections are post/get methods which shouldn't take > >>> very long at all. > >>> > >>> CPU/MEM/JVM all appear to be within normal operating > >>> limits. I've not had much luck searching for articles > >>> for this behavior nor finding remedies. The default > >>> timeout values are used in both Tomcat and in the > >>> applications that run within as far as I can tell. > >>> Hopefully someone will have some insight on why the > >>> behavior could be occurring, why isn't Tomcat killing > >>> the connections? Even in a RST/ACK status, shouldn't > >>> Tomcat terminate the connection without an ACK from the > >>> client after the default timeout? > > Can you please post: > > 1. Complete Tomcat version > > I can't find anything more granular than 9.0.29, is there > > a command to show a sub patch level? > > > > 9.0.29 is the patch-level, so that's fine. You are about 10 > > versions out of date (~1 year). Any chance for an upgrade? > > > >> They had to re-dev many apps last year when we upgraded from I > >> want to say 1 or 3 or something equally as horrific. Hopefully > >> they are forward compatible with the newer releases and if not > >> should surely be tackled now before later, I will certainly bring > >> this to the table! > > I've rarely been bitten by an upgrade from foo.bar.x to foo.bar.y. > There is a recent caveat if you are using the AJP connector, but you > are not so it's not an issue for you. > > 2. Connector configuration (possibly redacted) > > This is the 8443 section of the server.xml *8080 is > > available during the outage and I'm able to curl the > > management page to see the 300 used threads, their status, > > and age* > > > > [snip] > > > > > connectionTimeout="2" redirectPort="8443" /> [snip] > > > protocol="org.apache.coyote.http11.Http11NioProtocol" > > maxThreads="300" SSLEnabled="true" > > > > certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > > > > > certificateKeystorePassword="redacted" type="RSA" /> > > [snip] > port="8443" > > protocol="org.apache.coyote.http11.Http11NioProtocol" > > maxThreads="300" SSLEnabled="true" > > protocols="TLSv1.2"> > certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > > > > > certificateKeystorePassword="redacted" type="RSA" /> > > > > > > What, two connectors on one port? Do you get errors when starting? > >> No errors, one is "with HTTP2" should I delete the other former? > > Well, one of them will succeed in starting the and other one should > fail. Did you copy/paste your config without modification? Weird you > don't have any errors. Usually you'll get an IOException or whatever > binding to the port twice. I do recall IOExceptions and "port already in use" errors that caused Tomcat to not start, but I think these were related to syntax errors when defining catalina variables for my JVM sizing. I'll take another look at catalina.out and ensure I don't still see these, and will likely clean up the non "with http2" connector out of the config regardless. The only edits to the section of the supplied xml were the .jks store name and pw. > > > I don't see anything obviously problematic in the above > > configuration (other than the double-definition of the 8443 > > connector). > > > > 300 tied-up connections (from your initial report) sounds like a > > significant number: probably the thread count. > >> Yes sir, that's the NIO thread count for the 8443 connector. > > > > Mark (as is often the case) is right: take some thread dumps next > > time everything locks up and see what all those threads are doing. > > Often, it's something like everything is awaiting on a db > > connection and the db pool has been exhausted or something. > > Relatively simple quick-fixes are available for that, and better, > >
Re: Probelm with shutdown script
On Thu, Aug 27, 2020, 16:16 Christopher Schultz < ch...@christopherschultz.net> wrote: [ snip ] If you want to *kill* the application and it won't shut down on its > own, SIGKILL is the answer. But that's not a great way to shut down an > application /in general/ because the application might want/need to do > something convenient on shutdown (flush caches, save state, etc.). SIGTERM is the polite way to ask an application to shut down. A JVM will respond to this signal. (SIGHUP will also initiate a shut down, but TERM is preferred). Using TERM will cause the shutdown hooks to initiate. As Chris states, SIGKILL is a last resort (hooks are not called). [ snip ]
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 8/27/20 17:14, David wrote: > Thank you all for the replies! > > On Thu, Aug 27, 2020 at 3:53 PM Christopher Schultz > wrote: >> > David, > > On 8/27/20 13:57, David wrote: On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz wrote: > David, On 8/27/20 10:48, David wrote: >>> In the last two weeks I've had two occurrences where a >>> single CentOS 7 production server hosting a public >>> webpage has become unresponsive. The first time, all >>> 300 available "https-jsse-nio-8443" threads were >>> consumed, with the max age being around 45minutes, and >>> all in a "S" status. This time all 300 were consumed in >>> "S" status with the oldest being around ~16minutes. A >>> restart of Tomcat on both occasions freed these threads >>> and the website became responsive again. The >>> connections are post/get methods which shouldn't take >>> very long at all. >>> >>> CPU/MEM/JVM all appear to be within normal operating >>> limits. I've not had much luck searching for articles >>> for this behavior nor finding remedies. The default >>> timeout values are used in both Tomcat and in the >>> applications that run within as far as I can tell. >>> Hopefully someone will have some insight on why the >>> behavior could be occurring, why isn't Tomcat killing >>> the connections? Even in a RST/ACK status, shouldn't >>> Tomcat terminate the connection without an ACK from the >>> client after the default timeout? Can you please post: 1. Complete Tomcat version > I can't find anything more granular than 9.0.29, is there > a command to show a sub patch level? > > 9.0.29 is the patch-level, so that's fine. You are about 10 > versions out of date (~1 year). Any chance for an upgrade? > >> They had to re-dev many apps last year when we upgraded from I >> want to say 1 or 3 or something equally as horrific. Hopefully >> they are forward compatible with the newer releases and if not >> should surely be tackled now before later, I will certainly bring >> this to the table! I've rarely been bitten by an upgrade from foo.bar.x to foo.bar.y. There is a recent caveat if you are using the AJP connector, but you are not so it's not an issue for you. 2. Connector configuration (possibly redacted) > This is the 8443 section of the server.xml *8080 is > available during the outage and I'm able to curl the > management page to see the 300 used threads, their status, > and age* > > [snip] > > connectionTimeout="2" redirectPort="8443" /> [snip] > protocol="org.apache.coyote.http11.Http11NioProtocol" > maxThreads="300" SSLEnabled="true" > > certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > > certificateKeystorePassword="redacted" type="RSA" /> > [snip] port="8443" > protocol="org.apache.coyote.http11.Http11NioProtocol" > maxThreads="300" SSLEnabled="true" > protocols="TLSv1.2"> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > > certificateKeystorePassword="redacted" type="RSA" /> > > > What, two connectors on one port? Do you get errors when starting? >> No errors, one is "with HTTP2" should I delete the other former? Well, one of them will succeed in starting the and other one should fail. Did you copy/paste your config without modification? Weird you don't have any errors. Usually you'll get an IOException or whatever binding to the port twice. > I don't see anything obviously problematic in the above > configuration (other than the double-definition of the 8443 > connector). > > 300 tied-up connections (from your initial report) sounds like a > significant number: probably the thread count. >> Yes sir, that's the NIO thread count for the 8443 connector. > > Mark (as is often the case) is right: take some thread dumps next > time everything locks up and see what all those threads are doing. > Often, it's something like everything is awaiting on a db > connection and the db pool has been exhausted or something. > Relatively simple quick-fixes are available for that, and better, > longer-term fixes as well. > >> Mark/Chris is there a way to dump the connector threads >> specifically? Or simply is it all contained as a machine/process >> thread? Sorry I'm not really a Linux guy. Most of the threads in the server will be connector threads. They will have names like https-nio-[port]-exec-[number]. If you get a thread dump[1], you'll get a stack trace from every thread. Rainer wrote a great presentation about them in the context of Tomcat. Feel free to give it a read: http://home.apache.org/~rjung/presentations/2018-06-13-ApacheRoadShow-Ja vaThreadDumps.pdf Do you have a single F5 or a group of them? > A group of them, several HA pairs depending on internal or > external
Re: Probelm with shutdown script
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Roger, On 8/27/20 14:43, Roger Marquis wrote: > Mark Thomas wrote: >> Those are all application issues. The application should shut >> itself down cleanly. Tomcat is complaining because it hasn't. > > I don't know Mark, most Java/Tomcat engineers expect an application > to shutdown when it's os/container/shell/parent shuts-down. Can > you help us understand why Tomcat is different in this respect than > say Apache httpd? Along with Mark's reply, there is also the fact that httpd doesn't really run "applications" per se. If you are using something like mod_php or CGI, then that's all ephemeral: once the request is done, the child process dies. Or, more specifically, the child process exits and that's what causes the response to be sent to the client. There is little to no notion of cross-request state in httpd. Requesting httpd to stop sends a signal to the primary httpd process which tells all its children to stop. If there is a problem with a child (e.g. it's processing a request, or stuck in some way), the primary process has to decide what to do about it. It may be able to SIGKILL it. Maybe not, depending upon process privileges (though the primary process usually runs as root, so it should be able to do that). For threaded MPMs, the primary or child process can choose what to do about killing the threads that appear stuck. The pthreads library has a pthread_cancel function which requests that the thread be stopped, but it does not guarantee anything. This is similar to Java, except that Java has gone so far as to say "the Thread.stop method should not be used" (because it is not reliable, it causes weirdness in the JVM, etc.). If you want to *kill* the application and it won't shut down on its own, SIGKILL is the answer. But that's not a great way to shut down an application /in general/ because the application might want/need to do something convenient on shutdown (flush caches, save state, etc.). The previous two paragraphs are mostly valid for both httpd AND Tomcat. The reason httpd is often "cleaner" than Tomcat sometimes is, is because of the applications running within Tomcat. httpd is really a web server, and Tomcat is an application container. Could Tomcat be more aggressive about killing applications when they aren't behaving appropriately? Maybe. But then you'd never fix your application, now would you? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9IInAACgkQHPApP6U8 pFi6zw//WBFH087QqzmZbgdQjPxDhZ5f5L9lQmp4VFaw+dOEkP4xNMkGCxWeTq1Z 1nmogFYsZ8vns8z1f8RJ22aDWhdFqCDDHlJkqobR/4d4wiKd0hmjm75s8NH2YeCk 2UwwlStV99n5BNu3/TBTQFyjUv2rhz8cO8tDkztFGS7avXogSm08zeeHgVIC8cWV tiGBe7QVH/PMeanGl+/1IdWQ5PmpmNpGBv5YGJMYfU9Hmul/GeNW55pLJPSxS7ll /8thgxDVK4VudnA6MGELVSht56qlEQtlekMJjSUUvagmT94yTDXd7teOagZL9U79 26EL5tOiSkE4GGPoR/jQa0kj0bNMHMUJsBJRoHDCXxQdJhPLTGZNKZM1yMgAlMnc 6D2/acg9TbO6ia0Q1qpQmh2vWUlMPdp4U8X1uP6RdKKQyyjM5Pzinw/ysktgEBpI lYSI+uz3uG0iJGzqECPaoTpTTtSqdtG68QVqU2Kgv4lHmENNHFWMDKbB3glwqfBN KLw6fCrKGG6XGdQEcgtR8GjyNNpErLcf7PzIqXtwo9bNoQWNhYF/VjjndRJTTL98 IwoEL9qHXs0hhtKHA4x7kir0OARjwXDNwuJGu9LrKX1qOtzNxUAmjckHtLDjb5m8 nWmsz2zzy1rak1t1EPVsFjfOu5kZL+++IQcXja29uZshuiPcWbI= =vc24 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
Thank you all for the replies! On Thu, Aug 27, 2020 at 3:53 PM Christopher Schultz wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > David, > > On 8/27/20 13:57, David wrote: > > On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz > > wrote: > >> > > David, > > > > On 8/27/20 10:48, David wrote: > In the last two weeks I've had two occurrences where a > single CentOS 7 production server hosting a public webpage > has become unresponsive. The first time, all 300 available > "https-jsse-nio-8443" threads were consumed, with the max age > being around 45minutes, and all in a "S" status. This time > all 300 were consumed in "S" status with the oldest being > around ~16minutes. A restart of Tomcat on both occasions > freed these threads and the website became responsive again. > The connections are post/get methods which shouldn't take > very long at all. > > CPU/MEM/JVM all appear to be within normal operating limits. > I've not had much luck searching for articles for this > behavior nor finding remedies. The default timeout values are > used in both Tomcat and in the applications that run within > as far as I can tell. Hopefully someone will have some > insight on why the behavior could be occurring, why isn't > Tomcat killing the connections? Even in a RST/ACK status, > shouldn't Tomcat terminate the connection without an ACK from > the client after the default timeout? > > > > Can you please post: > > > > 1. Complete Tomcat version > >> I can't find anything more granular than 9.0.29, is there a > >> command to show a sub patch level? > > 9.0.29 is the patch-level, so that's fine. You are about 10 versions > out of date (~1 year). Any chance for an upgrade? They had to re-dev many apps last year when we upgraded from I want to say 1 or 3 or something equally as horrific. Hopefully they are forward compatible with the newer releases and if not should surely be tackled now before later, I will certainly bring this to the table! > > > 2. Connector configuration (possibly redacted) > >> This is the 8443 section of the server.xml *8080 is available > >> during the outage and I'm able to curl the management page to see > >> the 300 used threads, their status, and age* >> name="Catalina"> > >> > >> [snip] > >> > >> >> connectionTimeout="2" redirectPort="8443" /> [snip] > >> >> protocol="org.apache.coyote.http11.Http11NioProtocol" > >> maxThreads="300" SSLEnabled="true" > > >> >> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > >> certificateKeystorePassword="redacted" type="RSA" /> > >> [snip] >> protocol="org.apache.coyote.http11.Http11NioProtocol" > >> maxThreads="300" SSLEnabled="true" > >> protocols="TLSv1.2"> >> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > >> certificateKeystorePassword="redacted" type="RSA" /> > >> > > What, two connectors on one port? Do you get errors when starting? No errors, one is "with HTTP2" should I delete the other former? > > I don't see anything obviously problematic in the above configuration > (other than the double-definition of the 8443 connector). > > 300 tied-up connections (from your initial report) sounds like a > significant number: probably the thread count. Yes sir, that's the NIO thread count for the 8443 connector. > > Mark (as is often the case) is right: take some thread dumps next time > everything locks up and see what all those threads are doing. Often, > it's something like everything is awaiting on a db connection and the > db pool has been exhausted or something. Relatively simple quick-fixes > are available for that, and better, longer-term fixes as well. > Mark/Chris is there a way to dump the connector threads specifically? Or simply is it all contained as a machine/process thread? Sorry I'm not really a Linux guy. > > Do you have a single F5 or a group of them? > >> A group of them, several HA pairs depending on internal or > >> external and application. This server is behind one HA pair and > >> is a single server. > > Okay. Just remember that each F5 can make some large number of > connections to Tomcat, so you need to make sure you can handle them. > > This was a much bigger deal back in the BIO days when thread limit = > connection limit, and the thread limit was usually something like 250 > - - 300. NIO is much better, and the default connection limit is 10k > which "ought to be enough for anyone"[1]. (lol) I'm more used to the 1-1 of the BIO style, which kinda confused me when I asked the F5 to truncate >X connections and alert me and there were 600+ connections while Tomcat manager stated ~30. Then I read what the non-interrupt was about. > > > > [1] With apologies to Bill gates, who apparently never said anything > of the sort. Thanks again, David > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEE
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Felix, On 8/27/20 16:09, Felix Schumacher wrote: > > Am 27.08.20 um 19:35 schrieb Christopher Schultz: >> David, >> >> On 8/27/20 10:48, David wrote: >>> In the last two weeks I've had two occurrences where a single >>> CentOS 7 production server hosting a public webpage has become >>> unresponsive. The first time, all 300 available >>> "https-jsse-nio-8443" threads were consumed, with the max age >>> being around 45minutes, and all in a "S" status. This time all >>> 300 were consumed in "S" status with the oldest being around >>> ~16minutes. A restart of Tomcat on both occasions freed these >>> threads and the website became responsive again. The >>> connections are post/get methods which shouldn't take very long >>> at all. >> >>> CPU/MEM/JVM all appear to be within normal operating limits. >>> I've not had much luck searching for articles for this behavior >>> nor finding remedies. The default timeout values are used in >>> both Tomcat and in the applications that run within as far as I >>> can tell. Hopefully someone will have some insight on why the >>> behavior could be occurring, why isn't Tomcat killing the >>> connections? Even in a RST/ACK status, shouldn't Tomcat >>> terminate the connection without an ACK from the client after >>> the default timeout? >> >> Can you please post: >> >> 1. Complete Tomcat version 2. Connector configuration (possibly >> redacted) >> >>> Is there a graceful way to script the termination of threads >>> in case Tomcat isn't able to for whatever reason? >> >> Not really. > > (First look at Marks response on determining the root cause) > > Well, there might be a way (if it is sane, I don't know). You can > configure a valve to look for seemingly stuck threads and try to > interrupt them: > > http://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Stuck_Thread _Detection_Valve > > There are a few caveats there. First it is only working, when > both conditions are true > > * the servlets are synchronous * the stuck thread can be "freed" > with an Interrupt > > But really, if your threads are stuck for more than 15 minutes, you > have ample of time to take a thread dump and hopefully find the > root cause, so that you don't need this valve. This is a good idea as a band-aid, but the reality is that if you need the StuckThreadDetectionValve then your application is probably broken and needs to be fixed. Here are things that can be broken which might cause thread exhaustion: 1. Poor resource management. Things like db connections pools which can leak and/or not be refilled by the application. Everything stops when the db pool dries up. 2. Failure to set proper IO timeouts. Guess what the default read timeout is on a socket? Forever! If you read from a socket you might never hear back. Sounds like a problem. Set your read timeouts, kids. You might need to do this on your HTTP connections (and pools, and factories, and connection-wrappers like Apache http-client), your database config (usually in the config URL), and any remote-API libraries you are using (which use e.g. HTTP under the hood). - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9IICoACgkQHPApP6U8 pFgkuQ/+NE7tC+wWXoP2Ntv0yljJHyasRY/3dVewoNUxfO4CwcEkhSpK5YEkiHd3 sE7jygxEn3SHtHJ0WQPBWMAzL9RoLnglbAsxVXuWCzbQzd3tGCKOxQevCN3y+2ft jffqMEqOCgrN4kvKivj75V3alFQotT+jbZm1nJEwuQCLSJCqiHWcyCLlJF9Y6axn Thvsv40bnTKCPgqezo/0AYiYjQ9xIatTC3QDw129E7bofNKPBLk7LWcbg9CQBu+T iboA8IIxFgrOFYn66Mgx4kcJcQTRJ2XgdJ1v8p+mSITWH3UkLa5OhZeTqU6x2LDl LPuY8eC6y9QUqpFeEtaL72ZpDdYAn7Vcu4B3+D4Oobh7o2EJNQijIQ6A2QKIFw6e eBACKL0JJMwvfxVnp3nKIuoB3yOemMGZ8NpqUNcEn5mjmZubRWXXJXjtjjF5pGYW RRbMXvs3tFhLGsqnjVHQ/AV5MyuYKfl4Tqhvrz0u2oh0A8uo5Kq3CuHBDcLhLjs1 RkDiZuSdVugRFeq6hcQAyqO6LQ/QRhqtQ1hscecr9Iv8grvs8gzi4PvlurgBFqEF AuWe0V2WY0IJ9S7BqUUDr3Ij+0CQgxeQ70yyOztWjsT0B6ZPdOChm5Meu1+qi2ni EuT6Q5Lo2KHTqhrvi/RbTUXs+D6LSNFN6QFOzWtKWAr+gyrQjKg= =Ew/J -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 8/27/20 13:57, David wrote: > On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz > wrote: >> > David, > > On 8/27/20 10:48, David wrote: In the last two weeks I've had two occurrences where a single CentOS 7 production server hosting a public webpage has become unresponsive. The first time, all 300 available "https-jsse-nio-8443" threads were consumed, with the max age being around 45minutes, and all in a "S" status. This time all 300 were consumed in "S" status with the oldest being around ~16minutes. A restart of Tomcat on both occasions freed these threads and the website became responsive again. The connections are post/get methods which shouldn't take very long at all. CPU/MEM/JVM all appear to be within normal operating limits. I've not had much luck searching for articles for this behavior nor finding remedies. The default timeout values are used in both Tomcat and in the applications that run within as far as I can tell. Hopefully someone will have some insight on why the behavior could be occurring, why isn't Tomcat killing the connections? Even in a RST/ACK status, shouldn't Tomcat terminate the connection without an ACK from the client after the default timeout? > > Can you please post: > > 1. Complete Tomcat version >> I can't find anything more granular than 9.0.29, is there a >> command to show a sub patch level? 9.0.29 is the patch-level, so that's fine. You are about 10 versions out of date (~1 year). Any chance for an upgrade? > 2. Connector configuration (possibly redacted) >> This is the 8443 section of the server.xml *8080 is available >> during the outage and I'm able to curl the management page to see >> the 300 used threads, their status, and age* > name="Catalina"> >> >> [snip] >> >> > connectionTimeout="2" redirectPort="8443" /> [snip] >> > protocol="org.apache.coyote.http11.Http11NioProtocol" >> maxThreads="300" SSLEnabled="true" > >> > certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" >> certificateKeystorePassword="redacted" type="RSA" /> >> [snip] > protocol="org.apache.coyote.http11.Http11NioProtocol" >> maxThreads="300" SSLEnabled="true" > > protocols="TLSv1.2"> > certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" >> certificateKeystorePassword="redacted" type="RSA" /> >> What, two connectors on one port? Do you get errors when starting? I don't see anything obviously problematic in the above configuration (other than the double-definition of the 8443 connector). 300 tied-up connections (from your initial report) sounds like a significant number: probably the thread count. Mark (as is often the case) is right: take some thread dumps next time everything locks up and see what all those threads are doing. Often, it's something like everything is awaiting on a db connection and the db pool has been exhausted or something. Relatively simple quick-fixes are available for that, and better, longer-term fixes as well. > Do you have a single F5 or a group of them? >> A group of them, several HA pairs depending on internal or >> external and application. This server is behind one HA pair and >> is a single server. Okay. Just remember that each F5 can make some large number of connections to Tomcat, so you need to make sure you can handle them. This was a much bigger deal back in the BIO days when thread limit = connection limit, and the thread limit was usually something like 250 - - 300. NIO is much better, and the default connection limit is 10k which "ought to be enough for anyone"[1]. - -chris [1] With apologies to Bill gates, who apparently never said anything of the sort. -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9IHUYACgkQHPApP6U8 pFgcMhAAsN/Fc0nG4EJ/aaxZtj46g7FW2UDLa3HcGI+r8mvI5pYlCxWO0Cm4oDHn PAEUsjNgDcyLbWPa+hIfTWZ2v594w8ACrprpdNWHoPhZ316LudpG3G8RWwrIVsOa pn6MmX79rvds1Htl2cbsZkaaNCg/3+dx5VgyQtexHopSP9FpU1swDwex4fIf/pFz jcl4SB6eMnKxHwf/avwEy6sfdN05ALCl6KfJBCA6vnRvMT8hYVGs5B/bDdPRU5zL 0cNIAlNaxrcw0G13cuOhg5KYG+eeKBKl2R/OiSmyn4+Xp7zzbl3G3i4GvfbYrwqe BFTcTGT3cTE3vwMcHmsskh2soxYcH3etWtJ2/XsrKoKdRqXpWybVyNEvHcUwhxdP h7SAN5V8D2+9a8Hhh8y/hUCHBOT70THUyBipYweV26wUj4ipOAiYiJ2UaCATwNzf E7Giv6D4Y9WQCU119HaQ65TLmvGTtfzctM5pJzrnRbI7LOpuo9/bNYxkYNoU8U9r sAgY4t3kvKNttetFnwdY5+JTM+yrFolYFkYMFv8vpaVyiumP4+dpbkniRAmLabWl O0kIn/bRTkek4ic/qCuawBi1Rc1hV1/1uUE1+t8XHG7Sfdd0vwUabZ8ZRxNUBWcc EliCfzyMosWcsgU2puNduPyXDeRxKb5gfe4VdfaH5xvfdqIpfgw= =SesB -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
Am 27.08.20 um 19:35 schrieb Christopher Schultz: > David, > > On 8/27/20 10:48, David wrote: > > In the last two weeks I've had two occurrences where a single > > CentOS 7 production server hosting a public webpage has become > > unresponsive. The first time, all 300 available > > "https-jsse-nio-8443" threads were consumed, with the max age being > > around 45minutes, and all in a "S" status. This time all 300 were > > consumed in "S" status with the oldest being around ~16minutes. A > > restart of Tomcat on both occasions freed these threads and the > > website became responsive again. The connections are post/get > > methods which shouldn't take very long at all. > > > CPU/MEM/JVM all appear to be within normal operating limits. I've > > not had much luck searching for articles for this behavior nor > > finding remedies. The default timeout values are used in both > > Tomcat and in the applications that run within as far as I can > > tell. Hopefully someone will have some insight on why the behavior > > could be occurring, why isn't Tomcat killing the connections? Even > > in a RST/ACK status, shouldn't Tomcat terminate the connection > > without an ACK from the client after the default timeout? > > Can you please post: > > 1. Complete Tomcat version > 2. Connector configuration (possibly redacted) > > > Is there a graceful way to script the termination of threads in > > case Tomcat isn't able to for whatever reason? > > Not really. (First look at Marks response on determining the root cause) Well, there might be a way (if it is sane, I don't know). You can configure a valve to look for seemingly stuck threads and try to interrupt them: http://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Stuck_Thread_Detection_Valve There are a few caveats there. First it is only working, when both conditions are true * the servlets are synchronous * the stuck thread can be "freed" with an Interrupt But really, if your threads are stuck for more than 15 minutes, you have ample of time to take a thread dump and hopefully find the root cause, so that you don't need this valve. Felix > > > My research for killing threads results in system threads or > > application threads, not Tomcat Connector connection threads, so > > I'm not sure if this is even viable. I'm also looking into ways to > > terminate these aged sessions via the F5. At this time I'm open to > > any suggestions that would be able to automate a resolution to > > keep the system from experiencing downtime, or for any insight on > > where to look for a root cause. Thanks in advance for any guidance > > you can lend. > It might actually be the F5 itself, especially if it opens up a large > number of connections to Tomcat and then tries to open additional ones > for some reason. If it opens 300 connections (which are then e.g. > leaked by the F5 internally) but the 301st is refused, then your > server is essentially inert from that point forward. > > NIO connectors default to max 10k connections so that's not likely the > actual problem, here, but it could be for some configurations. > > Do you have a single F5 or a group of them? > > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Probelm with shutdown script
On 27/08/2020 19:43, Roger Marquis wrote: > Mark Thomas wrote: >> Those are all application issues. The application should shut itself >> down cleanly. Tomcat is complaining because it hasn't. > > I don't know Mark, most Java/Tomcat engineers expect an application to > shutdown when it's os/container/shell/parent shuts-down. Can you help > us understand why Tomcat is different in this respect than say Apache > httpd? Java doesn't provide an API a cleanly stop another thread. If an application starts a non-daemon thread but doesn't stop it there is nothing Tomcat can do to stop that thread and the JVM won't shutdown until the thread stops. Using OS level tools to stop the process is the only option. Hence, applications are responsible to cleaning up the resources they create. For example, if an app starts a thread then the app is also responsible for stopping it. Generally, apps use one or more ServletContextListeners to do this. Along similar lines, failing to clean up resources can also trigger memory leaks. Tomcat will clean those up where it can and log the problems it finds so the application can address them. While such issues are unimportant when shutting Tomcat down entirely, they are much more serious if the web application is being redeployed. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
On 27/08/2020 18:57, David wrote: > On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz > wrote: Is there a graceful way to script the termination of threads in case Tomcat isn't able to for whatever reason? > > Not really. What you can do is take a thread dump when this happens so you can see what the threads are doing. That should provide some insight to where the problem is. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache 8.5.57 shared class loader does not find its default classpath
Are you sure, that the Tomcat you reach under the ip and port is the same, than that you reach by dns? Have you checked, whether the Java version running Tomcat is new enough to read the class lib.Text? Are there any other errors in catalina.out or localhost.DATE.log in the Tomcat instance, that is throwing the error? Felix Am 27.08.20 um 20:34 schrieb Carles Franquesa: > Chris, > > Thank you very much for the help. Follows the $unzip -v aprenonline.war > output. > > I've put away a whole folder of sql sources that the war contains just to > make this output shorter. The reference to Text.class is in the sixth > position of WEB-INF files. > > This is it: > > Archive: aprenonline.war > Length Method Size Cmpr Date Time CRC-32 Name > -- --- -- - > 0 Stored 0 0% 2020-08-27 13:18 META-INF/ > 103 Stored 103 0% 2020-08-27 13:18 3d32040a > META-INF/MANIFEST.MF > 0 Stored 0 0% 2020-08-27 13:18 WEB-INF/ > 0 Stored 0 0% 2020-08-27 13:18 WEB-INF/classes/ > 0 Stored 0 0% 2020-08-27 13:18 > WEB-INF/classes/lib/ > 0 Stored 0 0% 2020-08-27 13:18 > WEB-INF/classes/model/ > 0 Stored 0 0% 2020-08-27 13:18 > WEB-INF/classes/servlets/ > 0 Stored 0 0% 2020-08-27 13:18 > WEB-INF/classes/servlets/ao/ > > 0 Stored 0 0% 2020-08-27 13:18 WEB-INF/lib/ > 0 Stored 0 0% 2020-08-27 13:18 ao/ > 0 Stored 0 0% 2020-08-27 13:18 ao/css/ > 0 Stored 0 0% 2020-08-27 13:18 > confirma_preinscripcions/ > 0 Stored 0 0% 2020-08-27 13:18 css/ > 0 Stored 0 0% 2020-08-27 13:18 css/dialegs/ > 0 Stored 0 0% 2020-08-27 13:18 css/main/ > 0 Stored 0 0% 2020-08-27 13:18 css/parts/ > 0 Stored 0 0% 2020-08-27 13:18 cursos/ > 0 Stored 0 0% 2020-08-27 13:18 estat/ > 0 Stored 0 0% 2020-08-27 13:18 img/ > 0 Stored 0 0% 2020-08-27 13:18 js/ > 0 Stored 0 0% 2020-08-27 13:18 js/jquery/ > 0 Stored 0 0% 2020-08-27 13:18 mail_conegut/ > 0 Stored 0 0% 2020-08-27 13:18 matriculacio/ > 0 Stored 0 0% 2020-08-27 13:18 nou_estudiant/ > 0 Stored 0 0% 2020-08-27 13:18 pagament/ > 0 Stored 0 0% 2020-08-27 13:18 verificacio/ > 92 Stored 92 0% 2020-08-27 13:18 722fe088 > META-INF/context.xml > 88 Stored 88 0% 2020-08-27 13:18 386832d5 > WEB-INF/classes/a.bat > 84 Stored 84 0% 2020-08-27 13:18 05546721 > WEB-INF/classes/l.bat > 3045 Stored 3045 0% 2020-08-27 13:18 49e914c6 > WEB-INF/classes/lib/Fitxer.class > 17744 Stored 17744 0% 2020-08-27 13:18 ff442cb9 > WEB-INF/classes/lib/Pagina.class > 6104 Stored 6104 0% 2020-08-27 13:18 76df9796 > WEB-INF/classes/lib/Registre.class > 3047 Stored 3047 0% 2020-08-27 13:18 34720d8b > WEB-INF/classes/lib/Text.class > 2679 Stored 2679 0% 2020-08-27 13:18 738d5f31 > WEB-INF/classes/lib/csv.class > 242 Stored 242 0% 2020-08-27 13:18 1052a3c9 > WEB-INF/classes/lib/lib.class > 1155 Stored 1155 0% 2020-08-27 13:18 3314a2b8 > WEB-INF/classes/lib/numeriques.class > 838 Stored 838 0% 2020-08-27 13:18 43515f3d > WEB-INF/classes/lib/sexe.class > 1682 Stored 1682 0% 2020-08-27 13:18 1e7a0936 > WEB-INF/classes/lib/temps.class > 6217 Stored 6217 0% 2020-08-27 13:18 d127aa85 > WEB-INF/classes/model/Connexio.class > 1876 Stored 1876 0% 2020-08-27 13:18 7fd4edf3 > WEB-INF/classes/model/curs.class > 1311 Stored 1311 0% 2020-08-27 13:18 fd7b55be > WEB-INF/classes/model/docent.class > 1658 Stored 1658 0% 2020-08-27 13:18 f085c9d9 > WEB-INF/classes/model/estudiant.class > 2404 Stored 2404 0% 2020-08-27 13:18 89836b06 > WEB-INF/classes/model/persona.class > 1012 Stored 1012 0% 2020-08-27 13:18 6604d075 > WEB-INF/classes/model/preinscripcio.class > 88 Stored 88 0% 2020-08-27 13:18 d5b1a89d > WEB-INF/classes/r.bat > 1400 Stored 1400 0% 2020-08-27 13:18 2e06d9bb > WEB-INF/classes/servlets/FileLocationContextListener.class > 6338 Stored 6338 0% 2020-08-27 13:18 8da94aec > WEB-INF/classes/servlets/UploadDownloadFileServlet.class > 1365 Stored 1365 0% 2020-08-27 13:18 8aa46dad > WEB-INF/classes/servlets/ao/accepta_pendent.class > 2650 Stored 2650 0% 2020-08-27 13:18 1b35e8ab > WEB-INF/classes/servlets/ao/acus_de_rebut.class > 2301
Re: Tomcat JDBC Pool Cleaner Deadlock Problem
Am 27.08.20 um 11:47 schrieb Gokhan Akgul: > Hi , > > I have been facing the deadlock issue for the last 2 months about > JDBCPoolCleaner Thread . > > Following config set in context.xml > > auth="Container" > type="javax.sql.DataSource" > factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" > driverClassName="com.mysql.jdbc.Driver" > > url="jdbc:mysql://adress:3306/db?useUnicode=true&characterEncoding=latin5&characterResultSet=latin5&zeroDateTimeBehavior=convertToNull&autoReconnect=true&interactiveClient=true" > username="user" > password="pass" > initialSize="10" > maxActive="30" > maxIdle="15" > minIdle="10" > maxWait="3" > timeBetweenEvictionRunsMillis="5000" > minEvictableIdleTimeMillis="6" > removeAbandonedTimeout="600" > removeAbandoned="true" > logAbandoned="false" > testWhileIdle="true" > testOnBorrow="true" > testOnReturn="false" > validationQuery="/* ping */ SELECT 1" > validationInterval="3" > jmxEnabled="true" > > jdbcInterceptors="ConnectionState;StatementFinalizer;ResetAbandonedTimer;SlowQueryReport" > /> > > > > Thread dump > > Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 state=BLOCKED > - waiting to lock <0x57dcb0b7> (a com.mysql.jdbc.JDBC4PreparedStatement) > owned by http-nio-8080-exec-8 id=25 So, Tomcat tries to close a connection, that it thinks is abandoned (i.e. in your setup hasn't been used for more than 600 seconds and the pool of idle connections is getting empty). The connection is still in use ... > at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078) > at > com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584) > at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364) > at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556) > at > org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:388) > at > org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.java:618) > at > org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java:612) > at > org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:569) > at > org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPool.java:999) > at > org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1468) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > > Locked synchronizers: count = 1 > - java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868 > > > > http-nio-8080-exec-8 id=25 state=BLOCKED > - waiting to lock <0x42de9995> (a com.mysql.jdbc.JDBC4Connection) > owned by Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 ... by hibernate. The question now is, how often is your healtheck getting called (every 10 min == 600s)? If they are close together, you might want to set the abandoned timeout higher than the healthcheck interval, or you could try to close the connection (or whatever the equivalent is in hibernate (a session?)) in your code. Felix > at > com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl.java:5435) > at > com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaData.java:3269) > at com.mysql.jdbc.DatabaseMetaData.(DatabaseMetaData.java:675) > at > com.mysql.jdbc.JDBC4DatabaseMetaData.(JDBC4DatabaseMetaData.java:39) > at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at com.mysql.jdbc.DatabaseMetaData.getInstance(DatabaseMetaData.java:657) > at com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3064) > at com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3056) > at > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2315) > at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210) > at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) > at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
Re: Probelm with shutdown script
Mark Thomas wrote: Those are all application issues. The application should shut itself down cleanly. Tomcat is complaining because it hasn't. I don't know Mark, most Java/Tomcat engineers expect an application to shutdown when it's os/container/shell/parent shuts-down. Can you help us understand why Tomcat is different in this respect than say Apache httpd? Roger Marquis - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache 8.5.57 shared class loader does not find its default classpath
Chris, Thank you very much for the help. Follows the $unzip -v aprenonline.war output. I've put away a whole folder of sql sources that the war contains just to make this output shorter. The reference to Text.class is in the sixth position of WEB-INF files. This is it: Archive: aprenonline.war Length MethodSize CmprDateTime CRC-32 Name -- --- -- - 0 Stored0 0% 2020-08-27 13:18 META-INF/ 103 Stored 103 0% 2020-08-27 13:18 3d32040a META-INF/MANIFEST.MF 0 Stored0 0% 2020-08-27 13:18 WEB-INF/ 0 Stored0 0% 2020-08-27 13:18 WEB-INF/classes/ 0 Stored0 0% 2020-08-27 13:18 WEB-INF/classes/lib/ 0 Stored0 0% 2020-08-27 13:18 WEB-INF/classes/model/ 0 Stored0 0% 2020-08-27 13:18 WEB-INF/classes/servlets/ 0 Stored0 0% 2020-08-27 13:18 WEB-INF/classes/servlets/ao/ 0 Stored0 0% 2020-08-27 13:18 WEB-INF/lib/ 0 Stored0 0% 2020-08-27 13:18 ao/ 0 Stored0 0% 2020-08-27 13:18 ao/css/ 0 Stored0 0% 2020-08-27 13:18 confirma_preinscripcions/ 0 Stored0 0% 2020-08-27 13:18 css/ 0 Stored0 0% 2020-08-27 13:18 css/dialegs/ 0 Stored0 0% 2020-08-27 13:18 css/main/ 0 Stored0 0% 2020-08-27 13:18 css/parts/ 0 Stored0 0% 2020-08-27 13:18 cursos/ 0 Stored0 0% 2020-08-27 13:18 estat/ 0 Stored0 0% 2020-08-27 13:18 img/ 0 Stored0 0% 2020-08-27 13:18 js/ 0 Stored0 0% 2020-08-27 13:18 js/jquery/ 0 Stored0 0% 2020-08-27 13:18 mail_conegut/ 0 Stored0 0% 2020-08-27 13:18 matriculacio/ 0 Stored0 0% 2020-08-27 13:18 nou_estudiant/ 0 Stored0 0% 2020-08-27 13:18 pagament/ 0 Stored0 0% 2020-08-27 13:18 verificacio/ 92 Stored 92 0% 2020-08-27 13:18 722fe088 META-INF/context.xml 88 Stored 88 0% 2020-08-27 13:18 386832d5 WEB-INF/classes/a.bat 84 Stored 84 0% 2020-08-27 13:18 05546721 WEB-INF/classes/l.bat 3045 Stored 3045 0% 2020-08-27 13:18 49e914c6 WEB-INF/classes/lib/Fitxer.class 17744 Stored17744 0% 2020-08-27 13:18 ff442cb9 WEB-INF/classes/lib/Pagina.class 6104 Stored 6104 0% 2020-08-27 13:18 76df9796 WEB-INF/classes/lib/Registre.class 3047 Stored 3047 0% 2020-08-27 13:18 34720d8b WEB-INF/classes/lib/Text.class 2679 Stored 2679 0% 2020-08-27 13:18 738d5f31 WEB-INF/classes/lib/csv.class 242 Stored 242 0% 2020-08-27 13:18 1052a3c9 WEB-INF/classes/lib/lib.class 1155 Stored 1155 0% 2020-08-27 13:18 3314a2b8 WEB-INF/classes/lib/numeriques.class 838 Stored 838 0% 2020-08-27 13:18 43515f3d WEB-INF/classes/lib/sexe.class 1682 Stored 1682 0% 2020-08-27 13:18 1e7a0936 WEB-INF/classes/lib/temps.class 6217 Stored 6217 0% 2020-08-27 13:18 d127aa85 WEB-INF/classes/model/Connexio.class 1876 Stored 1876 0% 2020-08-27 13:18 7fd4edf3 WEB-INF/classes/model/curs.class 1311 Stored 1311 0% 2020-08-27 13:18 fd7b55be WEB-INF/classes/model/docent.class 1658 Stored 1658 0% 2020-08-27 13:18 f085c9d9 WEB-INF/classes/model/estudiant.class 2404 Stored 2404 0% 2020-08-27 13:18 89836b06 WEB-INF/classes/model/persona.class 1012 Stored 1012 0% 2020-08-27 13:18 6604d075 WEB-INF/classes/model/preinscripcio.class 88 Stored 88 0% 2020-08-27 13:18 d5b1a89d WEB-INF/classes/r.bat 1400 Stored 1400 0% 2020-08-27 13:18 2e06d9bb WEB-INF/classes/servlets/FileLocationContextListener.class 6338 Stored 6338 0% 2020-08-27 13:18 8da94aec WEB-INF/classes/servlets/UploadDownloadFileServlet.class 1365 Stored 1365 0% 2020-08-27 13:18 8aa46dad WEB-INF/classes/servlets/ao/accepta_pendent.class 2650 Stored 2650 0% 2020-08-27 13:18 1b35e8ab WEB-INF/classes/servlets/ao/acus_de_rebut.class 2301 Stored 2301 0% 2020-08-27 13:18 578e8ce2 WEB-INF/classes/servlets/ao/envia.class 3010 Stored 3010 0% 2020-08-27 13:18 d6df56a1 WEB-INF/classes/servlets/ao/fes_csv.class 4169 Stored 4169 0% 2020-08-27 13:18 444e6ec3 WEB-INF/classes/servlets/ao/puja_adjunts.class 1904 Stored 1904 0% 2020-08-27 13:18 6d8b0ed8 WEB-INF/classes/servlets/ao/update_curs.class 1885 Stored 1885 0% 2020-08-27 13:18 14479ab9 WEB-INF/classes/servlets/check_login.class 3951 Stored 3951 0% 2020-08-27 13:18 e1fade2c WEB
Re: Tomcat JDBC Pool Cleaner Deadlock Problem
I had a similar issue with JDBC and it got fixed by adding this parameter 'numTestsPerEvictionRun'=> '3', 'auth' => 'Container', 'driverClassName' => 'oracle.jdbc.OracleDriver', 'maxActive' => '30', 'maxIdle' => '8', 'maxWait' => '-1', 'minIdle' => '1', 'factory' => 'org.apache.tomcat.jdbc.pool.DataSourceFactory', 'testWhileIdle' => 'true', 'testOnReturn' => 'false', 'validationQuery' => 'SELECT 1 FROM DUAL', 'testOnBorrow' => 'true', 'maxWait' => '-1', 'initialSize' => '1', 'removeAbandoned' => 'true', 'logAbandoned' => 'true', 'jmxEnabled'=> 'true', 'jdbcInterceptors' => 'org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer', 'validationInterval'=> '3', 'timeBetweenEvictionRunsMillis' => '3', 'minEvictableIdleTimeMillis'=> '3', 'removeAbandonedTimeout'=> '60', 'numTestsPerEvictionRun'=> '3', On Thu, Aug 27, 2020 at 1:44 PM Christopher Schultz wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Gokhan, > > On 8/27/20 05:47, Gokhan Akgul wrote: > > Hi , > > > > I have been facing the deadlock issue for the last 2 months about > > JDBCPoolCleaner Thread . > > > > Following config set in context.xml > > > > > type="javax.sql.DataSource" > > factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" > > driverClassName="com.mysql.jdbc.Driver" > > url="jdbc:mysql://adress:3306/db?useUnicode=true&characterEncoding > =latin5&characterResultSet=latin5&zeroDateTimeBehavior=convertTo > Null&autoReconnect=true&interactiveClient=true" > > > > > username="user" > > password="pass" initialSize="10" maxActive="30" maxIdle="15" > > minIdle="10" maxWait="3" timeBetweenEvictionRunsMillis="5000" > > minEvictableIdleTimeMillis="6" removeAbandonedTimeout="600" > > removeAbandoned="true" logAbandoned="false" testWhileIdle="true" > > testOnBorrow="true" testOnReturn="false" validationQuery="/* ping > > */ SELECT 1" validationInterval="3" jmxEnabled="true" > > jdbcInterceptors="ConnectionState;StatementFinalizer;ResetAbandonedTim > er;SlowQueryReport" > > > > > /> > > > > > > > > Thread dump > > > > Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 > > state=BLOCKED - waiting to lock <0x57dcb0b7> (a > > com.mysql.jdbc.JDBC4PreparedStatement) owned by > > http-nio-8080-exec-8 id=25 at > > com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078 > ) > > > > > at > com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java > :1584) > > at > > com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364) > > at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556) at > > org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnecti > on.java:388) > > > > > at > org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.ja > va:618) > > at > > org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java > :612) > > > > > at > org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:5 > 69) > > at > > org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPo > ol.java:999) > > > > > at > org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPoo > l.java:1468) > > at java.util.TimerThread.mainLoop(Timer.java:555) at > > java.util.TimerThread.run(Timer.java:505) > > > > Locked synchronizers: count = 1 - > > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868 > > > > > > > > > > http-nio-8080-exec-8 id=25 state=BLOCKED - waiting to lock > > <0x42de9995> (a com.mysql.jdbc.JDBC4Connection) owned by Tomcat > > JDBC Pool Cleaner[63445188:1598345711425] id=16 at > > com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl. > java:5435) > > > > > at > com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaDat > a.java:3269) > > at > > com.mysql.jdbc.DatabaseMetaData.(DatabaseMetaData.java:675) > > at > > com.mysql.jdbc.JDBC4DatabaseMetaData.(JDBC4DatabaseMetaData.java > :39) > > > > > at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source > ) > > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo > nstructorAccessorImpl.java:45) > > > > > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at > > com.mysql.jdbc.DatabaseMetaData.getInstance(DatabaseMetaData.j
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > David, > > On 8/27/20 10:48, David wrote: > > In the last two weeks I've had two occurrences where a single > > CentOS 7 production server hosting a public webpage has become > > unresponsive. The first time, all 300 available > > "https-jsse-nio-8443" threads were consumed, with the max age being > > around 45minutes, and all in a "S" status. This time all 300 were > > consumed in "S" status with the oldest being around ~16minutes. A > > restart of Tomcat on both occasions freed these threads and the > > website became responsive again. The connections are post/get > > methods which shouldn't take very long at all. > > > > CPU/MEM/JVM all appear to be within normal operating limits. I've > > not had much luck searching for articles for this behavior nor > > finding remedies. The default timeout values are used in both > > Tomcat and in the applications that run within as far as I can > > tell. Hopefully someone will have some insight on why the behavior > > could be occurring, why isn't Tomcat killing the connections? Even > > in a RST/ACK status, shouldn't Tomcat terminate the connection > > without an ACK from the client after the default timeout? > > Can you please post: > > 1. Complete Tomcat version I can't find anything more granular than 9.0.29, is there a command to show a sub patch level? > 2. Connector configuration (possibly redacted) This is the 8443 section of the server.xml *8080 is available during the outage and I'm able to curl the management page to see the 300 used threads, their status, and age* > > > Is there a graceful way to script the termination of threads in > > case Tomcat isn't able to for whatever reason? > > Not really. > > > My research for killing threads results in system threads or > > application threads, not Tomcat Connector connection threads, so > > I'm not sure if this is even viable. I'm also looking into ways to > > terminate these aged sessions via the F5. At this time I'm open to > > any suggestions that would be able to automate a resolution to > > keep the system from experiencing downtime, or for any insight on > > where to look for a root cause. Thanks in advance for any guidance > > you can lend. > It might actually be the F5 itself, especially if it opens up a large > number of connections to Tomcat and then tries to open additional ones > for some reason. If it opens 300 connections (which are then e.g. > leaked by the F5 internally) but the 301st is refused, then your > server is essentially inert from that point forward. > > NIO connectors default to max 10k connections so that's not likely the > actual problem, here, but it could be for some configurations. > > Do you have a single F5 or a group of them? A group of them, several HA pairs depending on internal or external and application. This server is behind one HA pair and is a single server. > > - -chris Thank you Chris! David > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H7tIACgkQHPApP6U8 > pFjR1hAAldbVnHDrV0W4aPLvcDwO/zi7qvrCscNjnJWhmR1m9TrevlrSb0EZvCJo > gTl7DXYEiZ9sBEdgs6AavHlk8jQ+ZbXbp8lsMElW5X9QoxxUD3YyJEpDSeHOG7/S > 2CyCYGzAQER0RlzBn9w97bCKWvUWoWDeLApd/pwdATjAo53hDtdNGdz6WdNLEzKm > g/BCZP0ynHZu7pMzSeZsOUBUXEKhDwHU+71fJo+WIJ4Gtiyb4xf2qkewvjQtuOGl > o/yESHNCJy09JAs3xK9W6eEVp981/Fuo4qDH32MJaXXbZRaNk32AwqngXKUhTW2l > BBl0jHoFIj+YJYc6AgVlv0la5qDIqP2VTv4ujOLBeF/95oP4sVRobIN4TiFyH6vv > ImvvRq55ML5NvKJv8g9Tj0aY5PusfwxcwyMCVovIof49vQXJUy7SbtgRB3eqgT8W > WwdBiGNsyWZpVjpr/CGBkBZmuR4wckeq0J5O/XGRFS9pK1jXH4gPnxe58vJmjA+P > RiSdp3SsU0P94SuF843CW+vmWyUu6SApCybUTwo5yiFXP2e/1+M9/fUGsykXpszU > zUvMcj9LWJ1QR3TbvEnwilsge4HKbUM3ZsFaujDjAVy6TAOgGS/dtVZ2UyMcrlOd > JMe3GeaOdM+ej27l5D8Eq6jaQcCfy+Mg9HxsYsbyrgrw3AhBhmo= > =eVIu > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat JDBC Pool Cleaner Deadlock Problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Gokhan, On 8/27/20 05:47, Gokhan Akgul wrote: > Hi , > > I have been facing the deadlock issue for the last 2 months about > JDBCPoolCleaner Thread . > > Following config set in context.xml > > type="javax.sql.DataSource" > factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" > driverClassName="com.mysql.jdbc.Driver" > url="jdbc:mysql://adress:3306/db?useUnicode=true&characterEncoding =latin5&characterResultSet=latin5&zeroDateTimeBehavior=convertTo Null&autoReconnect=true&interactiveClient=true" > > username="user" > password="pass" initialSize="10" maxActive="30" maxIdle="15" > minIdle="10" maxWait="3" timeBetweenEvictionRunsMillis="5000" > minEvictableIdleTimeMillis="6" removeAbandonedTimeout="600" > removeAbandoned="true" logAbandoned="false" testWhileIdle="true" > testOnBorrow="true" testOnReturn="false" validationQuery="/* ping > */ SELECT 1" validationInterval="3" jmxEnabled="true" > jdbcInterceptors="ConnectionState;StatementFinalizer;ResetAbandonedTim er;SlowQueryReport" > > /> > > > > Thread dump > > Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 > state=BLOCKED - waiting to lock <0x57dcb0b7> (a > com.mysql.jdbc.JDBC4PreparedStatement) owned by > http-nio-8080-exec-8 id=25 at > com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078 ) > > at com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java :1584) > at > com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364) > at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556) at > org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnecti on.java:388) > > at org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.ja va:618) > at > org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java :612) > > at org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:5 69) > at > org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPo ol.java:999) > > at org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPoo l.java:1468) > at java.util.TimerThread.mainLoop(Timer.java:555) at > java.util.TimerThread.run(Timer.java:505) > > Locked synchronizers: count = 1 - > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868 > > > > > http-nio-8080-exec-8 id=25 state=BLOCKED - waiting to lock > <0x42de9995> (a com.mysql.jdbc.JDBC4Connection) owned by Tomcat > JDBC Pool Cleaner[63445188:1598345711425] id=16 at > com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl. java:5435) > > at com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaDat a.java:3269) > at > com.mysql.jdbc.DatabaseMetaData.(DatabaseMetaData.java:675) > at > com.mysql.jdbc.JDBC4DatabaseMetaData.(JDBC4DatabaseMetaData.java :39) > > at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source ) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo nstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at > com.mysql.jdbc.DatabaseMetaData.getInstance(DatabaseMetaData.java:657) > > at com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3064) > at > com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3056) > > at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:231 5) > at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess orImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementP roxy.invoke(AbstractQueryReport.java:210) > > at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) > at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess orImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementP roxy.invoke(AbstractQueryReport.java:210) > > at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) > at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess orImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(Stat ementFacade.java:114) > > at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) > at > org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:2 08) > > at org.hibernate.loader.Loader.getResultSet(Loader.java:1953) > at org.hibernate.loader.Loader.doQuery(Loader.java:802) at > org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loa der.java:274) > > at org.hibernate.loader.Loader.doList(Loader.java:2542) > at > org.hibernate.loader.L
Re: Apache 8.5.57 shared class loader does not find its default classpath
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Carles,
On 8/27/20 12:19, Carles Franquesa wrote:
> Hi Everybody!, Just got in the list :)
>
> I am developing a webapp with Netbeans 8.0.2, and deploying it as a
> WAR file with Apache 8.5.57 Tomcat Manager onto my VPS where a
> mydomain.com is publically mapped on the DNS.
>
> It works fine in localhost, and even at the VPS when the IP and
> path is set in the url browser: http://ip:port/myapp. Then, it
> works.
>
> When trying to connect via my registered domain in the browser
> url, astonishingly, the index.jsp is correctly shown, but none of
> its links to other JSPs are going on. The first one is called
> cursos.jsp.
Do you mean that the links don't work (the browser won't follow them),
or you get an error when you click on those links because of the JSP
compilation errors?
> The included file before the tag is the same in index.jsp as
> in cursos.jsp, located in webapps/myapp/cursos/cursos.jsp which
> produces the error.
Your attachment was stripped from your message, but I don't think it
is really necessary to understand what's going on.
> The begining of both files is:
>
> --
- -
>
>
<%@page session="true" %>
> <%@page import="lib.Text"%>
> --
-
>
>
I also have been looking at stackoverflow, and found some amazing
> solutions, like ending the import with a semicolon. But it neither
> worked at all.
>
> My purpose is to make it work on mydomain.eu that I have already
> registered and mapped in the DNS.
>
> The error message given by any browser is printed next.
> --
-
>
>
Tipo Informe de Excepción
>
> mensaje Unable to compile class for JSP
>
> Descripción El servidor encontró un error interno que hizo que no
> pudiera rellenar este requerimiento.
>
> excepción
>
> org.apache.jasper.JasperException: Unable to compile class for
> JSP:
>
> An error occurred at line: [14] in the generated java file:
> [/opt/tomcat/work/Catalina/
> mydoamin.com/cursos/org/apache/jsp/cursos_jsp.java] Only a type can
> be imported. lib.Text resolves to a package
>
> ... and here, error is repeated for several classes
> --
-
>
> So, it seems that the class loader does not find the classes
> located at its default repository.
>
> I have been looking for the way tomcat uses classpaths. In
> particular, the role of its class loaders. And specifically, the
> "shared.loader" class loader.
>
> I even tried to set the value "webapps/myapp/WEB-INF/classes" in
> the catalina.properties file (even though it is not supposed to be
> needed).
>
> I have been looking for this in
> http://tomcat.apache.org/tomcat-8.5-doc/class-loader-howto.html.
> There, learned about the four loaders of tomcat. And as long as I
> understood, the classes repository should be located at
> ${CATALINA_HOME}/webapps/myapp/WEB-INF/classes. The place where
> Tomcat Manager leaves it when the WAR is uploaded.
>
> Ultimately, the
> ${CATALINA_HOME}/webapps/myapp/WEB-INF/classes/lib/Text.class
> certainly exists!, so I am stuck in telling the loader where the
> classes are.
You shouldn't have to mess around with class loaders or anything,
though that was good information for you to read. And you have
understood it correctly!
Can you take the WAR file you have deployed on the server and run this
on it?
$ unzip -v my.war
Can you post the contents? It should contain, among other things:
WEB-INF/classes/lib/Text.class
I'm wondering if the WAR file isn't including something that is
present on your local system when you test (and where it works properly)
.
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H8DcACgkQHPApP6U8
pFg2FA/9Hxn/yJOxRWch+8tAYZsLvcuCF+aynod0BdDso1wyhVWmv44JDNgwcDbA
Q6VQH0KclheiGlZmfH59y1RTxeNUUYOw22wbr7qyoKq/ShsDxvmEEiRdq1hDrqVS
v6gnb7XMrtYIhPhRJDnOhm+vD4KmK28szSvTFRXvTUaENjFBGVFHc8AkDldRy7Df
g9F/VUadiKuO2D/z7RxbiHzDYt3yCgGCAq9+6pch5LoUQ3W0Bmg6+NYXxdXylRQh
EDVd6vhxrc/kFqbTFcP7Wmc/HJ9Y3mYQ2AYANQRO/9tmSYjaXqNXrITczLsltbU5
Z6M/1pw3flTycGjEA6ycLBP3CNTKykB1Ygda3plf7Zsf+m3w/4Mt6RYbsGPzOis0
/E4o5QPpdnIfWcXjU6Ivgdtk6q1z5QFBNehsJscXtNK93Y5tEas5Z3ldLBRh1+ZK
Oe9mcyNY70rrtAvT+2/QVJoYV6Z0nZmEKti+wnDY2NCX7UoS72FqN1ENHhOM/uPj
2Gc9gY0t3tAxjWvKjhQi5b9LaqW6tSm3o3xhD109u2Fck9Gr5NOz6Lf9LHWkYxHn
fLPmzgic5vp6VyceWq/F+zwQElnHwELBz1LR1lrX3kCKmxsHqsqjvLp2HK7Wt4us
f67rukAYn1Gl4l3lCTorSpRXTKBXE8Wc9c4z02BftLAhWMpDQqA=
=nSg9
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: user
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 8/27/20 10:48, David wrote: > In the last two weeks I've had two occurrences where a single > CentOS 7 production server hosting a public webpage has become > unresponsive. The first time, all 300 available > "https-jsse-nio-8443" threads were consumed, with the max age being > around 45minutes, and all in a "S" status. This time all 300 were > consumed in "S" status with the oldest being around ~16minutes. A > restart of Tomcat on both occasions freed these threads and the > website became responsive again. The connections are post/get > methods which shouldn't take very long at all. > > CPU/MEM/JVM all appear to be within normal operating limits. I've > not had much luck searching for articles for this behavior nor > finding remedies. The default timeout values are used in both > Tomcat and in the applications that run within as far as I can > tell. Hopefully someone will have some insight on why the behavior > could be occurring, why isn't Tomcat killing the connections? Even > in a RST/ACK status, shouldn't Tomcat terminate the connection > without an ACK from the client after the default timeout? Can you please post: 1. Complete Tomcat version 2. Connector configuration (possibly redacted) > Is there a graceful way to script the termination of threads in > case Tomcat isn't able to for whatever reason? Not really. > My research for killing threads results in system threads or > application threads, not Tomcat Connector connection threads, so > I'm not sure if this is even viable. I'm also looking into ways to > terminate these aged sessions via the F5. At this time I'm open to > any suggestions that would be able to automate a resolution to > keep the system from experiencing downtime, or for any insight on > where to look for a root cause. Thanks in advance for any guidance > you can lend. It might actually be the F5 itself, especially if it opens up a large number of connections to Tomcat and then tries to open additional ones for some reason. If it opens 300 connections (which are then e.g. leaked by the F5 internally) but the 301st is refused, then your server is essentially inert from that point forward. NIO connectors default to max 10k connections so that's not likely the actual problem, here, but it could be for some configurations. Do you have a single F5 or a group of them? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H7tIACgkQHPApP6U8 pFjR1hAAldbVnHDrV0W4aPLvcDwO/zi7qvrCscNjnJWhmR1m9TrevlrSb0EZvCJo gTl7DXYEiZ9sBEdgs6AavHlk8jQ+ZbXbp8lsMElW5X9QoxxUD3YyJEpDSeHOG7/S 2CyCYGzAQER0RlzBn9w97bCKWvUWoWDeLApd/pwdATjAo53hDtdNGdz6WdNLEzKm g/BCZP0ynHZu7pMzSeZsOUBUXEKhDwHU+71fJo+WIJ4Gtiyb4xf2qkewvjQtuOGl o/yESHNCJy09JAs3xK9W6eEVp981/Fuo4qDH32MJaXXbZRaNk32AwqngXKUhTW2l BBl0jHoFIj+YJYc6AgVlv0la5qDIqP2VTv4ujOLBeF/95oP4sVRobIN4TiFyH6vv ImvvRq55ML5NvKJv8g9Tj0aY5PusfwxcwyMCVovIof49vQXJUy7SbtgRB3eqgT8W WwdBiGNsyWZpVjpr/CGBkBZmuR4wckeq0J5O/XGRFS9pK1jXH4gPnxe58vJmjA+P RiSdp3SsU0P94SuF843CW+vmWyUu6SApCybUTwo5yiFXP2e/1+M9/fUGsykXpszU zUvMcj9LWJ1QR3TbvEnwilsge4HKbUM3ZsFaujDjAVy6TAOgGS/dtVZ2UyMcrlOd JMe3GeaOdM+ej27l5D8Eq6jaQcCfy+Mg9HxsYsbyrgrw3AhBhmo= =eVIu -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Merka, On 8/27/20 06:32, Phoenix, Merka wrote: > I think what the Qualys scan is trying to flag is that the server > (Tomcat) is listening for both secured and unsecured traffic on > the _same_ TCP port when the server should be listening for just > one of the two options (unsecured or secured), not both. This actually might be a semi-legitimate test. If the client says "Our policy is to only communicate using encrypted connections" and the test says "well, here's a non-encrypted connection right here!" then it makes some small amount of sense. In that case, it's all driven by policy, and the policy can say "we have a carve-out for TLS handshake failures" and then allow that particular test to pass. > The error message returned by the Tomcat service, while certainly > helpful to the remote client, is returning more information than > it should (from a security-viewpoint). Not really. > If the default behavior for Tomcat is to return this "helpful" > message for misconfigured clients, would it be reasonable for the > Connector to have an option (attribute) for turning off this > feature and simply reject with a TLS error any unsecured traffic on > the port? This would address the security concern without imposing > too much "bloat" to the Tomcat side. I think this might actually be better/easier than implementing a redirect. It's a simple flag that says "return nice errors on plaintext-over-TLS connection attempts" (or similar) and the only thing that changes is that we STOP trying to be nice to the client if the setting is set to "fail" versus "be-nice". > For most other web servers (Apache httpd, NGINX, etc.) that offer > https, the normal behavior is that when an http client tries to > speak http to server expecting https, the client sees some garbled > text (the server's TLS response to the connection attempt). This wasn't the case for httpd for many years. I don't know what it does these days, but it used to reply with a nice "400 Bad Request" error just like Tomcat is doing. The difference is that httpd has rich configuration options to allow you to override that behavior. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H7ZkACgkQHPApP6U8 pFgOng/8DtMJkQc+MYLRm1iuCD7GZ2f2S7oQ59vFeCGAbmkiUjESvQ42QRGqIMy8 47giFc3ERm84DxyyyU7O/YDFxinOnCrC/v9A6RzpYKlZBSOq9Oy6732xTUAqGGIw +3QGPXvyjE2Vcg1iavW+7cUKN1Q9R1NGcDRRBpBL0KRQMA3NV9pxGU71/In8TQvi VZ51f7VTNXaJ2l+w6G23XBJkKQk3csFixevVzr4Xr56FLfqPxUc3m6QNu4BaHmgb c95hceV/N7tkR9yHdaRtahpZq0lhGqXbNXfqjf7kElSkRmeAZ3MSsdnFD4fBHThn xvz204xffSE71Z4W24W9gx23+Bg0y2EfPRo1CWC93rEvNRKMK6ILzLczTRA0w6QW 9zP1XC+VwC25LQOGFDgFukQVupPYiMoNSb6DRey5ZUhur6v25nevwbhM0QsAm/oO oZpreKaUMy+ZoixwGhaZ+UFiZRav7DRLSj85BjK9PqcP4VdPzFR9MarvMqLPxRoq YxL/jNet4L+29Z2tDkZv4gfGJqI7oWkfXUsBFBjj5JgXrqE94Q8PzAK87pLeU80y p3IL4krovHbu01j1fE3/aDotEvBu/wxWTCWze9+vL09a82PuTT2pihyCVqFuP9rS kP0DtVTfbaUMyD2dryjyw4q1NdLYht4y/HHkyU/3cPopCbxEopU= =4F4z -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Apache 8.5.57 shared class loader does not find its default classpath
Hi Everybody!, Just got in the list :)
I am developing a webapp with Netbeans 8.0.2, and deploying it as a WAR
file with Apache 8.5.57 Tomcat Manager onto my VPS where a mydomain.com is
publically mapped on the DNS.
It works fine in localhost, and even at the VPS when the IP and path is set
in the url browser: http://ip:port/myapp. Then, it works.
When trying to connect via my registered domain in the browser url,
astonishingly, the index.jsp is correctly shown, but none of its links to
other JSPs are going on. The first one is called cursos.jsp.
The included file before the tag is the same in index.jsp as in
cursos.jsp, located in webapps/myapp/cursos/cursos.jsp which produces the
error. The begining of both files is:
---
<%@page session="true" %>
<%@page import="lib.Text"%>
--
I also have been looking at stackoverflow, and found some amazing
solutions, like ending the import with a semicolon. But it neither worked
at all.
My purpose is to make it work on mydomain.eu that I have already registered
and mapped in the DNS.
The error message given by any browser is printed next.
--
Tipo Informe de Excepción
mensaje Unable to compile class for JSP
Descripción El servidor encontró un error interno que hizo que no pudiera
rellenar este requerimiento.
excepción
org.apache.jasper.JasperException: Unable to compile class for JSP:
An error occurred at line: [14] in the generated java file:
[/opt/tomcat/work/Catalina/
mydoamin.com/cursos/org/apache/jsp/cursos_jsp.java]
Only a type can be imported. lib.Text resolves to a package
... and here, error is repeated for several classes
--
So, it seems that the class loader does not find the classes located at its
default repository.
I have been looking for the way tomcat uses classpaths. In particular, the
role of its class loaders. And specifically, the "shared.loader" class
loader.
I even tried to set the value "webapps/myapp/WEB-INF/classes" in the
catalina.properties file (even though it is not supposed to be needed).
I have been looking for this in
http://tomcat.apache.org/tomcat-8.5-doc/class-loader-howto.html. There,
learned about the four loaders of tomcat. And as long as I understood, the
classes repository should be located at
${CATALINA_HOME}/webapps/myapp/WEB-INF/classes. The place where Tomcat
Manager leaves it when the WAR is uploaded.
Ultimately, the
${CATALINA_HOME}/webapps/myapp/WEB-INF/classes/lib/Text.class certainly
exists!, so I am stuck in telling the loader where the classes are.
Thanks for your time,
Any help will be much appreciated!
Carles
Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
In the last two weeks I've had two occurrences where a single CentOS 7 production server hosting a public webpage has become unresponsive. The first time, all 300 available "https-jsse-nio-8443" threads were consumed, with the max age being around 45minutes, and all in a "S" status. This time all 300 were consumed in "S" status with the oldest being around ~16minutes. A restart of Tomcat on both occasions freed these threads and the website became responsive again. The connections are post/get methods which shouldn't take very long at all. CPU/MEM/JVM all appear to be within normal operating limits. I've not had much luck searching for articles for this behavior nor finding remedies. The default timeout values are used in both Tomcat and in the applications that run within as far as I can tell. Hopefully someone will have some insight on why the behavior could be occurring, why isn't Tomcat killing the connections? Even in a RST/ACK status, shouldn't Tomcat terminate the connection without an ACK from the client after the default timeout? Is there a graceful way to script the termination of threads in case Tomcat isn't able to for whatever reason? My research for killing threads results in system threads or application threads, not Tomcat Connector connection threads, so I'm not sure if this is even viable. I'm also looking into ways to terminate these aged sessions via the F5.At this time I'm open to any suggestions that would be able to automate a resolution to keep the system from experiencing downtime, or for any insight on where to look for a root cause. Thanks in advance for any guidance you can lend. Thanks, David
Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys
On 27/08/2020 11:32, Phoenix, Merka wrote: > The error message returned by the Tomcat service, while certainly helpful to > the remote client, is returning more information than it should (from a > security-viewpoint). What, exactly, are the security concerns here? Your comment suggests there is some form of information disclosure vulnerability. What information? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat v9 - Insecure transport vulnerability reported by Qualys
-Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Thursday, 27 August, 2020 00:42 To: users@tomcat.apache.org Subject: Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys ... (from earlier in this thread) > On Wed, Aug 26, 2020 at 7:53 AM Pratik Shrestha wrote: > >> Thanks for reply, >> >> Hi Peter - it complains on port 8443 which belongs to Tomcat. >> >> Hi Mark - Yes. making HTTP request on HTTPS is wrong. But this security >> vulnerability is given to us by Qualys scan. It tries to post plain HTTP >> request on HTTPS port and then gets error message "Bad Request. This >> combination >> of host and port requires TLS." which is security loop hole for Qualys. ... >> > With HTTPD rewrite, whether or not the request is encrypted or sent to >> > the correct port can be detected and the request redirected as >> > appropriate. Maybe the same can be done with the rewrite valve used with >> > Tomcat. > This isn't currently possible with Tomcat because of detection of plain > text HTTP when TLS should be used (and the generation of the associated > response) is much, much earlier in the processing chain than the rewrite > valve. Since the server side is expecting a TLS connection, and the TLS session must be established first _before_ the HTTP request can be sent across that connection, wouldn't it be more logical to simply return a TLS error and reject the connection attempt since the TLS session setup failed. How and why is the server attempting to respond to the HTTP before the TLS session has been established? I think what the Qualys scan is trying to flag is that the server (Tomcat) is listening for both secured and unsecured traffic on the _same_ TCP port when the server should be listening for just one of the two options (unsecured or secured), not both. The error message returned by the Tomcat service, while certainly helpful to the remote client, is returning more information than it should (from a security-viewpoint). If the default behavior for Tomcat is to return this "helpful" message for misconfigured clients, would it be reasonable for the Connector to have an option (attribute) for turning off this feature and simply reject with a TLS error any unsecured traffic on the port? This would address the security concern without imposing too much "bloat" to the Tomcat side. For most other web servers (Apache httpd, NGINX, etc.) that offer https, the normal behavior is that when an http client tries to speak http to server expecting https, the client sees some garbled text (the server's TLS response to the connection attempt). Cheers! Simba Engineering - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat JDBC Pool Cleaner Deadlock Problem
Hi , I have been facing the deadlock issue for the last 2 months about JDBCPoolCleaner Thread . Following config set in context.xml Thread dump Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 state=BLOCKED - waiting to lock <0x57dcb0b7> (a com.mysql.jdbc.JDBC4PreparedStatement) owned by http-nio-8080-exec-8 id=25 at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078) at com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584) at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364) at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556) at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:388) at org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.java:618) at org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java:612) at org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:569) at org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPool.java:999) at org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1468) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Locked synchronizers: count = 1 - java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868 http-nio-8080-exec-8 id=25 state=BLOCKED - waiting to lock <0x42de9995> (a com.mysql.jdbc.JDBC4Connection) owned by Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 at com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl.java:5435) at com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaData.java:3269) at com.mysql.jdbc.DatabaseMetaData.(DatabaseMetaData.java:675) at com.mysql.jdbc.JDBC4DatabaseMetaData.(JDBC4DatabaseMetaData.java:39) at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.DatabaseMetaData.getInstance(DatabaseMetaData.java:657) at com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3064) at com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3056) at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2315) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210) at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210) at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114) at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208) at org.hibernate.loader.Loader.getResultSet(Loader.java:1953) at org.hibernate.loader.Loader.doQuery(Loader.java:802) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274) at org.hibernate.loader.Loader.doList(Loader.java:2542) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276) at org.hibernate.loader.Loader.list(Loader.java:2271) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:459) at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:365) at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268) at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:246) .. at com.gg.healthcheck.servlet.HealthCheckServlet.getServiceStatus(HealthCheckServlet.java:68) at com.gg.healthcheck.servlet.HealthCheckServlet.doGet(HealthCheckServlet.java:45) at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) at org.apache.catalina.co
Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys
Mark, Sorry for Top-posting. I’m still wondering what is causing this Qualys finding. I remember times when you got only garbage when you connected with http to https. Probably Qualys was fine with that. Now you get a nice 400 message that helps the user understand his mistake and Qualys jumps on that! From my point of view we should not change that behavior as it will not change the users settings or mistyping. I wonder how Nginx or httpd are reacting to this finding - if Qualys reacts in the same way? Basically the scanner already has the information that this is an SSL port! To me a bug in the scanner plugin! My 2ct. Peter > Am 27.08.2020 um 09:47 schrieb Mark Thomas : > > On 27/08/2020 06:31, Terence M. Bandoian wrote: >> On 8/26/2020 11:27 PM, Pratik Shrestha wrote: > > > >>> For me, there are two options for the fix which I am not able to make >>> them >>> work. >>> >>> 1. Either show 'ERR_EMPTY_RESP' like old Tomcat version 7 used to >>> show. As >>> far as I know, with Tomcat 7 giving that error, Qualys did not use to >>> show >>> this vulnerability. >>> 2. *Best is to do a redirect* when Tomcat sees error 400 to https URL. >>> Like >>> in Apache, we can add below. >>>'ErrorDocument 400 "*https*://xxx.xxx.xxx.xxx" >>> But as understood, redirect only works with error 3XX and ErrorDocument >>> feature is not there in Tomcat yet. > > > >> With HTTPD rewrite, whether or not the request is encrypted or sent to >> the correct port can be detected and the request redirected as >> appropriate. Maybe the same can be done with the rewrite valve used with >> Tomcat. > > This isn't currently possible with Tomcat because of detection of plain > text HTTP when TLS should be used (and the generation of the associated > response) is much, much earlier in the processing chain than the rewrite > valve. > > > > On 8/26/20 13:59, Mark Thomas wrote: >> On 26/08/2020 17:50, Christopher Schultz wrote: > > > >>> I'm interested in having Tomcat be able to pass these (admittedly >>> stupid) security requirements, >> I have no interest in adding bloat to Tomcat so it can pass so called >> security requirements that have no relevance to actual security. Those >> sort of changes are the sort that get me starting to think about using >> a veto. >> Understood. But what does the OP have in terms of options at this point? >> >> 1. Ignore the complaint (probably not possible) 2. Request a waiver for >> this issue (probably not possible, or at least would require 10 years of >> red tape) 3. Front the server with httpd + "ErrorDocument 400" (which >> ... I >> think will *also* reply with a plaintext response, right?) 4. Switch to >> Jetty> >> I'm trying to avoid "the easiest thing" which is probably to switch to >> Jetty. I know our "customers" don't pay for Tomcat, but losing a >> "customer" >> sucks. > > One of the things I love about working Tomcat is when this sort of > security nonsense comes along, I can a) call it out and b) veto (if I > have to) the implementation without someone higher up the organisational > hierarchy able to play the "I don't care if it is nonsense, our > customers want it so you have to implement it" card. > > My objection to implementing or changing features in response to > "security nonsense" is that it perpetuates the problem. If people who > know this is "security nonsense" just accept it rather than arguing > against it, that nonsense eventually becomes "security fact". I think > the world could do with a little more security fact and a little less > security nonsense. > > That said, I'm not against changing this feature where that change > offers real benefits to users. > >> How about being able to specify the response text, possibly blank? > > While I remember, there was the issue raised that the response wasn't > UTF-8 and we changed hard-coded response to UTF-8 rather than provide an > option. > > My concern with anything along the lines of making it configurable is > that because this response is generated outside of the normal HTTP > processing infrastructure you can quickly get into the situation where > you end up replicating functionality we already have elsewhere. > >> I think "ErrorDocument 400" with nothing else might mean the same >> thing as >> [[ErrorDocument 400 ""]] meaning that the response will include NO >> CONTENT. >> Maybe that's what Qualys is looking for. > > My reading of the thread so far is that the security scanner expects > either a TLS error or a redirect to https. > > The redirect option is interesting. I can see user benefit in http > requests to an https port getting redirected to https. The tricky part > is implementing it. Redirection to a fixed URL is quite simple. As soon > as you get into redirecting the actual user's request you open up the > HTTP request parsing can of worms. > > I'm wondering if there is a way to utilise the standard request > processing infrastructur
Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys
On 27/08/2020 06:31, Terence M. Bandoian wrote: > On 8/26/2020 11:27 PM, Pratik Shrestha wrote: >> For me, there are two options for the fix which I am not able to make >> them >> work. >> >> 1. Either show 'ERR_EMPTY_RESP' like old Tomcat version 7 used to >> show. As >> far as I know, with Tomcat 7 giving that error, Qualys did not use to >> show >> this vulnerability. >> 2. *Best is to do a redirect* when Tomcat sees error 400 to https URL. >> Like >> in Apache, we can add below. >> 'ErrorDocument 400 "*https*://xxx.xxx.xxx.xxx" >> But as understood, redirect only works with error 3XX and ErrorDocument >> feature is not there in Tomcat yet. > With HTTPD rewrite, whether or not the request is encrypted or sent to > the correct port can be detected and the request redirected as > appropriate. Maybe the same can be done with the rewrite valve used with > Tomcat. This isn't currently possible with Tomcat because of detection of plain text HTTP when TLS should be used (and the generation of the associated response) is much, much earlier in the processing chain than the rewrite valve. On 8/26/20 13:59, Mark Thomas wrote: > On 26/08/2020 17:50, Christopher Schultz wrote: >> I'm interested in having Tomcat be able to pass these (admittedly >> stupid) security requirements, > I have no interest in adding bloat to Tomcat so it can pass so called > security requirements that have no relevance to actual security. Those > sort of changes are the sort that get me starting to think about using > a veto. > Understood. But what does the OP have in terms of options at this point? > > 1. Ignore the complaint (probably not possible) 2. Request a waiver for > this issue (probably not possible, or at least would require 10 years of > red tape) 3. Front the server with httpd + "ErrorDocument 400" (which > ... I > think will *also* reply with a plaintext response, right?) 4. Switch to > Jetty> > I'm trying to avoid "the easiest thing" which is probably to switch to > Jetty. I know our "customers" don't pay for Tomcat, but losing a > "customer" > sucks. One of the things I love about working Tomcat is when this sort of security nonsense comes along, I can a) call it out and b) veto (if I have to) the implementation without someone higher up the organisational hierarchy able to play the "I don't care if it is nonsense, our customers want it so you have to implement it" card. My objection to implementing or changing features in response to "security nonsense" is that it perpetuates the problem. If people who know this is "security nonsense" just accept it rather than arguing against it, that nonsense eventually becomes "security fact". I think the world could do with a little more security fact and a little less security nonsense. That said, I'm not against changing this feature where that change offers real benefits to users. > How about being able to specify the response text, possibly blank? While I remember, there was the issue raised that the response wasn't UTF-8 and we changed hard-coded response to UTF-8 rather than provide an option. My concern with anything along the lines of making it configurable is that because this response is generated outside of the normal HTTP processing infrastructure you can quickly get into the situation where you end up replicating functionality we already have elsewhere. > I think "ErrorDocument 400" with nothing else might mean the same > thing as > [[ErrorDocument 400 ""]] meaning that the response will include NO > CONTENT. > Maybe that's what Qualys is looking for. My reading of the thread so far is that the security scanner expects either a TLS error or a redirect to https. The redirect option is interesting. I can see user benefit in http requests to an https port getting redirected to https. The tricky part is implementing it. Redirection to a fixed URL is quite simple. As soon as you get into redirecting the actual user's request you open up the HTTP request parsing can of worms. I'm wondering if there is a way to utilise the standard request processing infrastructure for these http over https requests (so we can use the existing http processing infrastructure to parse the request and issue a redirect to https) without creating the risk that we process the http request without the redirect as that would be very bad. Given the number of times redirect has been mentioned I think I'll spend some time looking into how possible it is to redirect the user's http request to https. My instinct is that it isn't going to be doable but it is worth a look. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
