Re: AW: AW: Having trouble with Tomcat crashes. Interesting memory numbers in Manager
I've tried profilers in the past, but I've never had much luck since you need a super computer to run them. Human intelligence rules .. read the code carefully, review it, step it with a debugger, and look for memory leak patterns. Mine have mostly been static and non static collections and HashMaps that keep growing, or rampant string creation wherein the GC couldn't keep up under load. Review the code .. become its master and empath. Memory leaks cause it pain. On 2/9/23, James H. H. Lampert wrote: > I've obtained some heap and CPU numbers, taking data at 15 minute > intervals, heap from WRKJVMJOB and CPU from WRKACTJOB. In two days of > this, I didn't witness any crashes; I did witness a near-miss, in which > heap-in-use hit 5011.938M (out of 5120). > > In discussion with our webapp developer (to whom we sent a catalina.out > excerpt), he observed that they were running Tomcat on a six-year-old > JVM (it identifies in a WRKJVMJOB as "1.8.0_151"; on the Manager page, > it identifies as "8.0.5.5 - pap6480sr5fp5-20171114_01(SR5 FP5)") with a > known issue (on Github, it's listed as 11493). He suggested that the > customer ought to try updating to a more recent Java. > > I've also asked on the IBM Midrange Java List whether we can go any > higher on the heap parameters (currently set at -Xms 4096 -Xmx 5120 for > that particular installation). > > -- > JHHL > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Basic SSL Certificate Usage logging
Once again, Awesome Possum! You guys are the greatest! Dream * Excel * Explore * Inspire Jon McAlexander Senior Infrastructure Engineer Asst. Vice President He/His Middleware Product Engineering Enterprise CIO | EAS | Middleware | Infrastructure Solutions 8080 Cobblestone Rd | Urbandale, IA 50322 MAC: F4469-010 Tel 515-988-2508 | Cell 515-988-2508 jonmcalexan...@wellsfargo.com This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. > -Original Message- > From: Mark Thomas > Sent: Friday, February 10, 2023 4:31 AM > To: users@tomcat.apache.org > Subject: Re: Basic SSL Certificate Usage logging > > On 09/02/2023 19:49, Christopher Schultz wrote: > > Jon, > > > > On 2/9/23 11:39, jonmcalexan...@wellsfargo.com.INVALID wrote: > >> My thinking is that the teams requesting that I look into if this is > >> possible or not would prefer to be able to get the more detailed > >> information if possible. How much extra work is required to have a > >> dedicated logger for it, as well as keeping the current message in > >> the current logging? > > It shouldn't be that much work, but it is a lot of output. > > +1 > > > Mark, isn't this already a dedicated logger? > > > > org.apache.tomcat.util.net.AbstractEndpoint.logCertificate > > Unfortunately not. That is the method name. > > > +1 to using the log-level as the arbiter for, well, how much logging > > +to > > do. :) > > :) > > I'll try and get this done in time for the next release round. > > Mark > > > > > -chris > > > >>> -Original Message- > >>> From: Mark Thomas > >>> Sent: Thursday, February 9, 2023 3:24 AM > >>> To: users@tomcat.apache.org > >>> Subject: Re: Basic SSL Certificate Usage logging > >>> > >>> Hi Jon, > >>> > >>> The current message looks like this: > >>> > >>> 09-Feb-2023 09:09:53.939 INFO [main] > >>> org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector > >>> [https- > >>> jsse-nio-8443], TLS virtual host [_default_], certificate type [RSA] > >>> configured from [conf/localhost-rsa.jks] using alias [tomcat] and > >>> with trust store [null] > >>> > >>> The intention is to make clear, for each configured server > >>> certificate, which configuration files are being used. The idea > >>> being that you can then examine the relevant files if there is an > >>> issue. > >>> > >>> There is a balance to strike in terms of providing useful detail and > >>> providing too much detail in the default logs. Everything below > >>> feels like too much detail. > >>> > >>> One option would be to switch this message to a dedicated logger and > >>> then provide more/all details if debug logging is enabled. Moving > >>> this to a dedicated logger would allow debug logging to be enabled > >>> for that logger without changing the logging for the entire > >>> endpoint. > >>> > >>> Mark > >>> > >>> > >>> On 08/02/2023 18:36, jonmcalexan...@wellsfargo.com.INVALID wrote: > Hi Mark, > > As a follow-up, some of my compatriots are asking if we can get all > or some > >>> of these details in the log as well? Wanted to ask early if possible. > > • Subject > o Ex: CN=splunk.glb.wellsfargo.net,OU=TMS-ADCS,O=Wells > >>> Fargo,C=US > o Ex: CN=9COM,OU=APP,OU=9COM,OU=ECS,O=Wells Fargo,C=US > o Ex: > CN=WFA-9CUS-PROD.wellsfargo.com,OU=9CUS,O=Wells > >>> Fargo,C=US > • SAN (aka Subject Alternative Names) o Ex: > DNS=splunk.wellsfargo.net;DNS=splunk.wellsfargo.com > o Ex: IP=170.43.135.39;DNS=nc-sils-dpb-znp10.wellsfargo.com; > o Ex: > >>> > EMAIL:some.n...@wellsfargo.com;EMAIL:some.name@wellsfargo.com > • Issuer > o Ex: CN=Wells Fargo Enterprise Certification Authority 05-2 > >>> G2,OU=Wells Fargo Certification Authorities,O=Wells Fargo,C=US > • ValidFrom (aka NotBefore) > o Ex: 2022-05-18T05:09:27Z > • ValidTo (aka NotAfter) > o Ex: 2024-05-17T05:09:27Z > • KeyUsage > o Ex: Digital Signature, Key Encipherment, Data Encipherment • > KeyUsageExtended o Ex: Client Authentication > (1.3.6.1.5.5.7.3.2), Server Authentication > >>> (1.3.6.1.5.5.7.3.1) > • SerialNumber > o Ex: 6a0006e41935f80460711c1876e419 > • FingerprintSHA1 (aka Thumbprint) o Ex: > 679323d7dcc9307d8696a88e0f1a8d4069a412b6 > • FingerprintSHA256 > o Ex: > >>> > DC5044B2E6A173CB2B05CEE54AA5B185DD6D4A341DC36B3CCB0DC99782DD4 > >>> E41 > • PublicKeyAlgo > o Ex: RSA > o Ex: ECDSA > • PublicKeySize > o Ex: 2048 > o Ex: P-256 > > Thank you, > > Dream *
RE: Tomcat 9.0.65 suspected memory leak
Thanks Mark, workaround seem to be working Chen > -Original Message- > From: Mark Thomas > Sent: Thursday, February 9, 2023 12:41 > To: users@tomcat.apache.org > Subject: Re: Tomcat 9.0.65 suspected memory leak > > On 09/02/2023 13:25, Mark Thomas wrote: > > On 09/02/2023 13:04, Mark Thomas wrote: > >> On 04/02/2023 22:06, Chen Levy wrote: > >> > >>> Mark, I believe a change in Tomcat 9.0.65 causes it to accumulate > >>> open connections: > >>> I took a fresh Tomcat, unzipped and modified server.xml with only > >>> the > >>> following: > >>> 1. Changed port 8080 to port 80 > >>> 2. Changed port 8443 to port 443 > >>> 3. Uncommented the nio connector and added the snippet > >>> >>> className="org.apache.coyote.http2.Http2Protocol" /> > >>> > >>> >>> certificateKeystoreFile="conf/tomcat_noroot.p12" > >>> certificateKeyAlias="..." > >>> certificateKeystorePassword="..." > >>> certificateKeystoreType="PKCS12"/> > >>> > >>> > >>> I used Chrome to call the default index.html with Wireshark in the > >>> middle: > >>> With 9.0.63 - 20 seconds after the last data frame, came a GOAWAY > >>> from the server. > >>> With 9.0.65 - No GOAWAY was sent, and the server and client kept > >>> ACKing each other. > >>> > >>> Tomcat 9.0.71 and 10.1.5 behaved similarly - no GOAWAY was sent. > >>> > >>> Test was conducted with: > >>> Wireshark Version 4.0.3 (v4.0.3-0-gc552f74cdc23) Chrome Version > >>> 109.0.5414.120 JDK 17.0.6+10 Windows 11 > >> > >> Thanks for the reproduction details. I'll take a look now. > > > > A quick workaround is to configure useAsyncIO="false" on the Connector. > > Fixed for the next round of releases. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Basic SSL Certificate Usage logging
On 09/02/2023 19:49, Christopher Schultz wrote: Jon, On 2/9/23 11:39, jonmcalexan...@wellsfargo.com.INVALID wrote: My thinking is that the teams requesting that I look into if this is possible or not would prefer to be able to get the more detailed information if possible. How much extra work is required to have a dedicated logger for it, as well as keeping the current message in the current logging? It shouldn't be that much work, but it is a lot of output. +1 Mark, isn't this already a dedicated logger? org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Unfortunately not. That is the method name. +1 to using the log-level as the arbiter for, well, how much logging to do. :) :) I'll try and get this done in time for the next release round. Mark -chris -Original Message- From: Mark Thomas Sent: Thursday, February 9, 2023 3:24 AM To: users@tomcat.apache.org Subject: Re: Basic SSL Certificate Usage logging Hi Jon, The current message looks like this: 09-Feb-2023 09:09:53.939 INFO [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https- jsse-nio-8443], TLS virtual host [_default_], certificate type [RSA] configured from [conf/localhost-rsa.jks] using alias [tomcat] and with trust store [null] The intention is to make clear, for each configured server certificate, which configuration files are being used. The idea being that you can then examine the relevant files if there is an issue. There is a balance to strike in terms of providing useful detail and providing too much detail in the default logs. Everything below feels like too much detail. One option would be to switch this message to a dedicated logger and then provide more/all details if debug logging is enabled. Moving this to a dedicated logger would allow debug logging to be enabled for that logger without changing the logging for the entire endpoint. Mark On 08/02/2023 18:36, jonmcalexan...@wellsfargo.com.INVALID wrote: Hi Mark, As a follow-up, some of my compatriots are asking if we can get all or some of these details in the log as well? Wanted to ask early if possible. • Subject o Ex: CN=splunk.glb.wellsfargo.net,OU=TMS-ADCS,O=Wells Fargo,C=US o Ex: CN=9COM,OU=APP,OU=9COM,OU=ECS,O=Wells Fargo,C=US o Ex: CN=WFA-9CUS-PROD.wellsfargo.com,OU=9CUS,O=Wells Fargo,C=US • SAN (aka Subject Alternative Names) o Ex: DNS=splunk.wellsfargo.net;DNS=splunk.wellsfargo.com o Ex: IP=170.43.135.39;DNS=nc-sils-dpb-znp10.wellsfargo.com; o Ex: EMAIL:some.n...@wellsfargo.com;EMAIL:some.name@wellsfargo.com • Issuer o Ex: CN=Wells Fargo Enterprise Certification Authority 05-2 G2,OU=Wells Fargo Certification Authorities,O=Wells Fargo,C=US • ValidFrom (aka NotBefore) o Ex: 2022-05-18T05:09:27Z • ValidTo (aka NotAfter) o Ex: 2024-05-17T05:09:27Z • KeyUsage o Ex: Digital Signature, Key Encipherment, Data Encipherment • KeyUsageExtended o Ex: Client Authentication (1.3.6.1.5.5.7.3.2), Server Authentication (1.3.6.1.5.5.7.3.1) • SerialNumber o Ex: 6a0006e41935f80460711c1876e419 • FingerprintSHA1 (aka Thumbprint) o Ex: 679323d7dcc9307d8696a88e0f1a8d4069a412b6 • FingerprintSHA256 o Ex: DC5044B2E6A173CB2B05CEE54AA5B185DD6D4A341DC36B3CCB0DC99782DD4 E41 • PublicKeyAlgo o Ex: RSA o Ex: ECDSA • PublicKeySize o Ex: 2048 o Ex: P-256 Thank you, Dream * Excel * Explore * Inspire Jon McAlexander Senior Infrastructure Engineer Asst. Vice President He/His Middleware Product Engineering Enterprise CIO | EAS | Middleware | Infrastructure Solutions 8080 Cobblestone Rd | Urbandale, IA 50322 MAC: F4469-010 Tel 515-988-2508 | Cell 515-988-2508 jonmcalexan...@wellsfargo.com This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: Mark Thomas Sent: Wednesday, February 8, 2023 10:37 AM To: users@tomcat.apache.org Subject: Re: Basic SSL Certificate Usage logging On 08/02/2023 16:24, jonmcalexan...@wellsfargo.com.INVALID wrote: Hi Mark, So, is this something that can/will be added in the future? I tested my thought of setting the java logging.properties to a specific file in the command line but it didn't do what I had hoped. Already added. Will be in the next round of releases. Mark Thanks, Dream * Excel * Explore * Inspire Jon McAlexander Senior Infrastructure Engineer Asst. Vice President He/His Middleware Product Engineering Enterprise CIO | EAS | Middleware | Infrastructure Solutions 8080 Cobblestone Rd | Urbandale, IA 50322 MAC: F4469-010 Tel 515-988-2508 | Cell 515-988-2508 jonmcalexan...@w
