Re: Log rotation issue
Hi Mark
Tomcat version
9.0.54
Operating system?
Linux- RedHat
Do you mean intermittent rather than intermediate?
Yes Intermittent if many errors or too much errors are getting dumped.
Usually when the same exception occurs multiple times.
This problem is seen.
What is writing to stdout? The "right" way to fix this is to stop
whatever is writing to stdout from doing that.
Mostly I think the exceptions and errors etc.,
From: Mark Thomas
Date: Monday, 27 February 2023 at 11:17 PM
To: users@tomcat.apache.org
Subject: Re: Log rotation issue
On 27/02/2023 02:28, Ragavendhiran Bhiman (rabhiman) wrote:
> Hi
>
> we are facing log rotation issue in apache tomcat. Our configurations like
> below
>
> handlers = 1catalina.org.apache.juli.AsyncFileHandler,
> 2localhost.org.apache.juli.AsyncFileHandler,
> 3manager.org.apache.juli.AsyncFileHandler,
> 4host-manager.org.apache.juli.AsyncFileHandler
>
> .handlers = 1catalina.org.apache.juli.AsyncFileHandler
>
>
> # Handler specific properties.
> # Describes specific configuration info for Handlers.
>
>
> 1catalina.org.apache.juli.AsyncFileHandler.level = FINE
> 1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
> 1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.
> 1catalina.org.apache.juli.AsyncFileHandler.rotatable = false
>
> 2localhost.org.apache.juli.AsyncFileHandler.level = FINE
> 2localhost.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
> 2localhost.org.apache.juli.AsyncFileHandler.prefix = localhost.
> 2localhost.org.apache.juli.AsyncFileHandler.rotatable = false
>
> 3manager.org.apache.juli.AsyncFileHandler.level = FINE
> 3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
> 3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
>
> 4admin.org.apache.juli.FileHandler.level = FINE
> 4admin.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
> 4admin.org.apache.juli.FileHandler.prefix = admin.
>
> 5host-manager.org.apache.juli.FileHandler.level = FINE
> 5host-manager.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
> 5host-manager.org.apache.juli.FileHandler.prefix = host-manager.
> java.util.logging.ConsoleHandler.level = FINE
> java.util.logging.ConsoleHandler.formatter = org.apache.juli.OneLineFormatter
>
>
>
> # Facility specific properties.
> # Provides extra control for each logger.
>
>
> org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
> org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers =
> 2localhost.org.apache.juli.AsyncFileHandler
>
> org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].level
> = INFO
> org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].handlers
> = 3manager.org.apache.juli.AsyncFileHandler
>
> org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].level
> = INFO
> org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].handlers
> = 4host-manager.org.apache.juli.AsyncFileHandler
>
> # For example, set the org.apache.catalina.util.LifecycleBase logger to log
> # each component that extends LifecycleBase changing state:
> #org.apache.catalina.util.LifecycleBase.level = FINE
>
> # To see debug messages in TldLocationsCache, uncomment the following line:
> #org.apache.jasper.compiler.TldLocationsCache.level = FINE
>
>
>
> It is an intermediate issue that the Catalina.out some times goes in GBs. How
> to control that?
Tomcat version?
Operating system?
Do you mean intermittent rather than intermediate?
What is writing to stdout? The "right" way to fix this is to stop
whatever is writing to stdout from doing that.
Short-term workarounds will depend on what operating system you are using.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Log rotation issue
Is using consolehandlers is fine?
From: Rob Sargent
Date: Monday, 27 February 2023 at 10:23 AM
To: users@tomcat.apache.org
Subject: Re: Log rotation issue
On 2/26/23 19:28, Ragavendhiran Bhiman (rabhiman) wrote:
> Hi
>
> we are facing log rotation issue in apache tomcat. Our configurations like
> below
>
> handlers = 1catalina.org.apache.juli.AsyncFileHandler,
> 2localhost.org.apache.juli.AsyncFileHandler,
> 3manager.org.apache.juli.AsyncFileHandler,
> 4host-manager.org.apache.juli.AsyncFileHandler
>
> .handlers = 1catalina.org.apache.juli.AsyncFileHandler
>
>
> # Handler specific properties.
> # Describes specific configuration info for Handlers.
>
>
> 1catalina.org.apache.juli.AsyncFileHandler.level = FINE
> 1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
> 1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.
> 1catalina.org.apache.juli.AsyncFileHandler.rotatable = false
>
> 2localhost.org.apache.juli.AsyncFileHandler.level = FINE
> 2localhost.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
> 2localhost.org.apache.juli.AsyncFileHandler.prefix = localhost.
> 2localhost.org.apache.juli.AsyncFileHandler.rotatable = false
>
> 3manager.org.apache.juli.AsyncFileHandler.level = FINE
> 3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
> 3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
>
> 4admin.org.apache.juli.FileHandler.level = FINE
> 4admin.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
> 4admin.org.apache.juli.FileHandler.prefix = admin.
>
> 5host-manager.org.apache.juli.FileHandler.level = FINE
> 5host-manager.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
> 5host-manager.org.apache.juli.FileHandler.prefix = host-manager.
> java.util.logging.ConsoleHandler.level = FINE
> java.util.logging.ConsoleHandler.formatter = org.apache.juli.OneLineFormatter
>
Have you tried change .level = FINE to .level = INFO?
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Log rotation issue
Hi
we are facing log rotation issue in apache tomcat. Our configurations like below
handlers = 1catalina.org.apache.juli.AsyncFileHandler,
2localhost.org.apache.juli.AsyncFileHandler,
3manager.org.apache.juli.AsyncFileHandler,
4host-manager.org.apache.juli.AsyncFileHandler
.handlers = 1catalina.org.apache.juli.AsyncFileHandler
# Handler specific properties.
# Describes specific configuration info for Handlers.
1catalina.org.apache.juli.AsyncFileHandler.level = FINE
1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.
1catalina.org.apache.juli.AsyncFileHandler.rotatable = false
2localhost.org.apache.juli.AsyncFileHandler.level = FINE
2localhost.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.AsyncFileHandler.prefix = localhost.
2localhost.org.apache.juli.AsyncFileHandler.rotatable = false
3manager.org.apache.juli.AsyncFileHandler.level = FINE
3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
4admin.org.apache.juli.FileHandler.level = FINE
4admin.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
4admin.org.apache.juli.FileHandler.prefix = admin.
5host-manager.org.apache.juli.FileHandler.level = FINE
5host-manager.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
5host-manager.org.apache.juli.FileHandler.prefix = host-manager.
java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = org.apache.juli.OneLineFormatter
# Facility specific properties.
# Provides extra control for each logger.
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers =
2localhost.org.apache.juli.AsyncFileHandler
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].level
= INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].handlers
= 3manager.org.apache.juli.AsyncFileHandler
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].level
= INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].handlers
= 4host-manager.org.apache.juli.AsyncFileHandler
# For example, set the org.apache.catalina.util.LifecycleBase logger to log
# each component that extends LifecycleBase changing state:
#org.apache.catalina.util.LifecycleBase.level = FINE
# To see debug messages in TldLocationsCache, uncomment the following line:
#org.apache.jasper.compiler.TldLocationsCache.level = FINE
It is an intermediate issue that the Catalina.out some times goes in GBs. How
to control that?
Kindly help to solve the issue.
Thanks & Regards,
Raghav
Hostheader attack vulnerability
Hi All, I am facing one issue related to host header manipulation changing the host header is chaning the url itself. This attack is done via the burp suite tool. I have copied the current configuration here as you could see the default hostname is defined and apBase is provided. The attack is happening only before the admin login page. Any pages displayed after the login the host header manipulation is not happening. Kindly advise me how to fix this problem from apache side.
Re: certificate re-loading for apache tomcat without the apache restart
Is there any way to reload new certificates as well with restarting the tomcat services? The mail below explains the modification of certificates only considered and not the new ones. Our scenario is to load new certificates as well if the nssdb got changed dynamically. Thanks & Regards, Raghavendran From: Ivano Luberti Date: Monday, 26 September 2022 at 12:51 PM To: users@tomcat.apache.org Subject: Re: certificate re-loading for apache tomcat without the apache restart Agree Here you can find documentation of what Peter says https://tomcat.apache.org/tomcat-10.0-doc/manager-howto.html#Reload_TLS_configuration using a call to the manager app. It doesn't take into account new certificates but only existing ones, because it dosn't reparse server.xml Il 26/09/2022 09:18, l...@kreuser.name ha scritto: > Raghavendran, > >> Am 26.09.2022 um 08:54 schrieb Ragavendhiran Bhiman >> (rabhiman): >> >> Hi All, >> >> I have a scenario where I need to reload the certificates which are newly >> updated in the NSS DB without restarting the apache – tomcat. >> Is there any way to do it? >> >> Kindly share some piece of code to achieve the reloading of the certificates >> without restarting the apache tomcat service itself. >> >> > curl -u -p > "https://myserver.mydomain/manager/jmxproxy?invoke=Catalina:type=ProtocolHandler,port= port>=reloadSslHostConfig=" > > you need that with at least roles="manager-jmx" in tomcat-users.xml > > >> Note : Trial from my side : Tried to restart the Apache connector, but still >> it is reloading the old certificates only and not the new certificates. >> If possible how to achieve the loading of the new one? >> >> >> Many Thanks for your help. >> >> Regards, >> >> Raghavendran >> > Hope this helps > > Peter -- Archimede Informatica tratta i dati personali in conformità a quanto stabilito dal Regolamento UE n. 2016/679 (GDPR) e dal D. Lgs. 30 giugno 2003 n. 196 per come modificato dal D.Lgs. 10 agosto 2018 n. 101. Informativa completa <http://www.archicoop.it/fileadmin/pdf/InformativaTrattamentoDatiPersonali.pdf> dott. Ivano Mario Luberti Archimede Informatica società cooperativa a r. l. Via Gereschi 36, 56127 Pisa tel.: +39 050/580959 | fax: +39 050/8932061 web: www.archicoop.it<http://www.archicoop.it> linkedin: www.linkedin.com/in/ivanoluberti<http://www.linkedin.com/in/ivanoluberti> facebook: www.facebook.com/archimedeinformaticapisa/<http://www.facebook.com/archimedeinformaticapisa/>
certificate re-loading for apache tomcat without the apache restart
Hi All, I have a scenario where I need to reload the certificates which are newly updated in the NSS DB without restarting the apache – tomcat. Is there any way to do it? Kindly share some piece of code to achieve the reloading of the certificates without restarting the apache tomcat service itself. Note : Trial from my side : Tried to restart the Apache connector, but still it is reloading the old certificates only and not the new certificates. If possible how to achieve the loading of the new one? Many Thanks for your help. Regards, Raghavendran
Re: AW: SSL handshake failure logs required for auditing purpose
Hi Christopher and all, Thanks for your great help. Raghav From: Christopher Schultz Date: Friday, 8 July 2022 at 9:28 PM To: users@tomcat.apache.org Subject: Re: AW: SSL handshake failure logs required for auditing purpose Raghav, On 7/8/22 06:36, Ragavendhiran Bhiman (rabhiman) wrote: > That’s great, and thank ful for your reply. > > Kindly look my below mail for my doubts, > > And need one more query can we have the same jar updated to 9.0.x lower > versions? > > If that particular jar is updated what is the jar? > > If jar is not possible what is the way we can get the solution to 9.0.x lower > versions. > > > Does via syslog this solution is possible? You can also get all of this using a network tap, sending it anywhere you want. No need to modify any software or even trust that the software is configured properly. Tomcat could be lying to you, and allowing failed connections from apache.org to be silently ignored. It just occurred to me that your request suggests that (your) Tomcat is being used directly by clients on the public internet. This is obviously a perfectly valid setup, but if I were running a publicly-accessible web-based application (which I do for $work), I would put something between the internet and my application to terminate TLS, perform load-balancing, etc. for a number of reasons. Is there any reason not to use another product that does *exactly* what you want, here? FWIW, I'm not sure if AWS/Azure/Oracle, httpd, nginx, squid, haproxy, etc. will report TLS handshake failures in a way that is acceptable to you and your certification body, either. I was just wondering... -chris > From: Ragavendhiran Bhiman (rabhiman) > Date: Friday, 8 July 2022 at 7:33 AM > To: Tomcat Users List > Subject: Re: AW: SSL handshake failure logs required for auditing purpose > Thanks a lot for all your replies. > > This auditing is for common criteria certification. The OS we use is Red-hat > Linux. > As you know common criteria requires these handshake failures need to be > redirected to a syslog server. > Any attempt via the tcp-dump/wireshark is not acceptable by the certification. > So it needs to be only the syslogs. > I think from 9.0.65 it should be easy. > For the existing versions yes the log needs to be in syslog until it rotates. > If it gives cipher details that’s good, but importantly it should give the > Ips. > > Once again thanks a lot for your overwhelming responses. If I will be able to > close this today, it is pretty great. > > Also let me know in 9.0.65 is there any detailed attempt made to log about > the ssl handshake including the ciphers etc.,? > > Regards, > > Raghav > > From: Christopher Schultz > Date: Friday, 8 July 2022 at 12:05 AM > To: users@tomcat.apache.org > Subject: Re: AW: SSL handshake failure logs required for auditing purpose > Thomas, > > On 7/7/22 13:36, Thomas Hoffmann (Speed4Trade GmbH) wrote: >> >> >>> -Ursprüngliche Nachricht- >>> Von: Thomas Hoffmann (Speed4Trade GmbH) >>> >>> Gesendet: Donnerstag, 7. Juli 2022 19:23 >>> An: Tomcat Users List >>> Betreff: AW: SSL handshake failure logs required for auditing purpose >>> >>> Hello Raghav, >>> >>>> -Ursprüngliche Nachricht- >>>> Von: Ragavendhiran Bhiman (rabhiman) >>>> Gesendet: Donnerstag, 7. Juli 2022 18:13 >>>> An: Tomcat Users List >>>> Betreff: Re: SSL handshake failure logs required for auditing purpose >>>> >>>> Version of tomcat used 9.0.x. >>>> Kindly help on the ssl logging for auditing purpose other than -D >>>> javax.net option. >>>> >>>> From: Ragavendhiran Bhiman (rabhiman) >>>> Date: Thursday, 7 July 2022 at 9:41 PM >>>> To: users@tomcat.apache.org >>>> Subject: SSL handshake failure logs required for auditing purpose Hi >>>> All, >>>> >>>> I require your kind help in logging the SSl connection failure logs >>>> including iP in the tomcat, Is there any best way to do It without >>>> performance impact other than -Djava.net debugs in jdk, is there any >>>> direct way from tomcat? Or any way we can derive any class from JSSE >>>> extension classes and add HandShakeListener while using the >>>> connectors. All our SSL connections are going through connectors. So >>>> kindly need your help how to log those SSL connection auditing logs >>> through best method. >>>> Thanks a lot in advance. >>>> >>>> Regards, >>>> Raghav >>> >>> Which OS are you using? >>> Can you use Wi
Re: AW: SSL handshake failure logs required for auditing purpose
Hi Mark, Thanks for your great help. Raghav From: Mark Thomas Date: Friday, 8 July 2022 at 4:44 PM To: users@tomcat.apache.org Subject: Re: AW: SSL handshake failure logs required for auditing purpose On 08/07/2022 11:36, Ragavendhiran Bhiman (rabhiman) wrote: > > > That’s great, and thank ful for your reply. > > Kindly look my below mail for my doubts, > > And need one more query can we have the same jar updated to 9.0.x lower > versions? No. The Apache Tomcat project does not produce patches for older versions. You are required to update to 9.0.65 or later. > If that particular jar is updated what is the jar? > > If jar is not possible what is the way we can get the solution to 9.0.x lower > versions. This is open source. You are free to try patching the code yourself. Personally, I'd judge that higher overall risk than updating. > Does via syslog this solution is possible? Yes, with a custom handler. e.g.: http://rusv.github.io/agafua-syslog/ (I've never used it, just found it via StackOverflow) > Thanks & Regards, > > Raghav > > From: Ragavendhiran Bhiman (rabhiman) > Date: Friday, 8 July 2022 at 7:33 AM > To: Tomcat Users List > Subject: Re: AW: SSL handshake failure logs required for auditing purpose > Thanks a lot for all your replies. > > This auditing is for common criteria certification. The OS we use is Red-hat > Linux. > As you know common criteria requires these handshake failures need to be > redirected to a syslog server. > Any attempt via the tcp-dump/wireshark is not acceptable by the certification. > So it needs to be only the syslogs. > I think from 9.0.65 it should be easy. > For the existing versions yes the log needs to be in syslog until it rotates. > If it gives cipher details that’s good, but importantly it should give the > Ips. > > Once again thanks a lot for your overwhelming responses. If I will be able to > close this today, it is pretty great. > > Also let me know in 9.0.65 is there any detailed attempt made to log about > the ssl handshake including the ciphers etc.,? You'll get the remote IP, remote port and whatever information is in the exception. https://github.com/apache/tomcat/blob/9.0.x/java/org/apache/tomcat/util/net/NioEndpoint.java#L1776 Mark > > Regards, > > Raghav > > From: Christopher Schultz > Date: Friday, 8 July 2022 at 12:05 AM > To: users@tomcat.apache.org > Subject: Re: AW: SSL handshake failure logs required for auditing purpose > Thomas, > > On 7/7/22 13:36, Thomas Hoffmann (Speed4Trade GmbH) wrote: >> >> >>> -Ursprüngliche Nachricht- >>> Von: Thomas Hoffmann (Speed4Trade GmbH) >>> >>> Gesendet: Donnerstag, 7. Juli 2022 19:23 >>> An: Tomcat Users List >>> Betreff: AW: SSL handshake failure logs required for auditing purpose >>> >>> Hello Raghav, >>> >>>> -Ursprüngliche Nachricht- >>>> Von: Ragavendhiran Bhiman (rabhiman) >>>> Gesendet: Donnerstag, 7. Juli 2022 18:13 >>>> An: Tomcat Users List >>>> Betreff: Re: SSL handshake failure logs required for auditing purpose >>>> >>>> Version of tomcat used 9.0.x. >>>> Kindly help on the ssl logging for auditing purpose other than -D >>>> javax.net option. >>>> >>>> From: Ragavendhiran Bhiman (rabhiman) >>>> Date: Thursday, 7 July 2022 at 9:41 PM >>>> To: users@tomcat.apache.org >>>> Subject: SSL handshake failure logs required for auditing purpose Hi >>>> All, >>>> >>>> I require your kind help in logging the SSl connection failure logs >>>> including iP in the tomcat, Is there any best way to do It without >>>> performance impact other than -Djava.net debugs in jdk, is there any >>>> direct way from tomcat? Or any way we can derive any class from JSSE >>>> extension classes and add HandShakeListener while using the >>>> connectors. All our SSL connections are going through connectors. So >>>> kindly need your help how to log those SSL connection auditing logs >>> through best method. >>>> Thanks a lot in advance. >>>> >>>> Regards, >>>> Raghav >>> >>> Which OS are you using? >>> Can you use Wireshark or TCPDump for your purposes? >>> If you are using Chrome or FF as Client, you can set the environment >>> variable >>> SSLKEYLOGFILE to write the current key to a file which Wireshark can take to >>> decrypt the traffic. >>> >>> The handshake itself is not encrypted. If the handshake is enough, TCPDump >
Re: AW: SSL handshake failure logs required for auditing purpose
That’s great, and thank ful for your reply. Kindly look my below mail for my doubts, And need one more query can we have the same jar updated to 9.0.x lower versions? If that particular jar is updated what is the jar? If jar is not possible what is the way we can get the solution to 9.0.x lower versions. Does via syslog this solution is possible? Thanks & Regards, Raghav From: Ragavendhiran Bhiman (rabhiman) Date: Friday, 8 July 2022 at 7:33 AM To: Tomcat Users List Subject: Re: AW: SSL handshake failure logs required for auditing purpose Thanks a lot for all your replies. This auditing is for common criteria certification. The OS we use is Red-hat Linux. As you know common criteria requires these handshake failures need to be redirected to a syslog server. Any attempt via the tcp-dump/wireshark is not acceptable by the certification. So it needs to be only the syslogs. I think from 9.0.65 it should be easy. For the existing versions yes the log needs to be in syslog until it rotates. If it gives cipher details that’s good, but importantly it should give the Ips. Once again thanks a lot for your overwhelming responses. If I will be able to close this today, it is pretty great. Also let me know in 9.0.65 is there any detailed attempt made to log about the ssl handshake including the ciphers etc.,? Regards, Raghav From: Christopher Schultz Date: Friday, 8 July 2022 at 12:05 AM To: users@tomcat.apache.org Subject: Re: AW: SSL handshake failure logs required for auditing purpose Thomas, On 7/7/22 13:36, Thomas Hoffmann (Speed4Trade GmbH) wrote: > > >> -Ursprüngliche Nachricht- >> Von: Thomas Hoffmann (Speed4Trade GmbH) >> >> Gesendet: Donnerstag, 7. Juli 2022 19:23 >> An: Tomcat Users List >> Betreff: AW: SSL handshake failure logs required for auditing purpose >> >> Hello Raghav, >> >>> -Ursprüngliche Nachricht- >>> Von: Ragavendhiran Bhiman (rabhiman) >>> Gesendet: Donnerstag, 7. Juli 2022 18:13 >>> An: Tomcat Users List >>> Betreff: Re: SSL handshake failure logs required for auditing purpose >>> >>> Version of tomcat used 9.0.x. >>> Kindly help on the ssl logging for auditing purpose other than -D >>> javax.net option. >>> >>> From: Ragavendhiran Bhiman (rabhiman) >>> Date: Thursday, 7 July 2022 at 9:41 PM >>> To: users@tomcat.apache.org >>> Subject: SSL handshake failure logs required for auditing purpose Hi >>> All, >>> >>> I require your kind help in logging the SSl connection failure logs >>> including iP in the tomcat, Is there any best way to do It without >>> performance impact other than -Djava.net debugs in jdk, is there any >>> direct way from tomcat? Or any way we can derive any class from JSSE >>> extension classes and add HandShakeListener while using the >>> connectors. All our SSL connections are going through connectors. So >>> kindly need your help how to log those SSL connection auditing logs >> through best method. >>> Thanks a lot in advance. >>> >>> Regards, >>> Raghav >> >> Which OS are you using? >> Can you use Wireshark or TCPDump for your purposes? >> If you are using Chrome or FF as Client, you can set the environment variable >> SSLKEYLOGFILE to write the current key to a file which Wireshark can take to >> decrypt the traffic. >> >> The handshake itself is not encrypted. If the handshake is enough, TCPDump >> or Wireshark are sufficient. >> >> Greetings, >> Thomas >> > > Short Addendum: > 1) Do you want to write the log permanently or just for an audit session? > 2) Which details do you want to log? Agreed cipher? Offered ciphers by the > client? SNI-header? ...? > 3) What is the purpose of the logging? > Insecure ciphers can be mitigated by server configuration. I think he wants to implement a poor-mans NIDS. Raghav, please be aware that any web browser that first attempts to use a SSLv3/TLSv1/TLSv1.3 handshake, fails, and retries with a TLSv1.2/similar handshake will cause massive numbers of false-positives in your logs. I would ask whoever is requesting this logging why they are looking at such failures. Handshake failures are not always indicative of some kind of intrusion attempt. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: SSL handshake failure logs required for auditing purpose
Thanks a lot for all your replies. This auditing is for common criteria certification. The OS we use is Red-hat Linux. As you know common criteria requires these handshake failures need to be redirected to a syslog server. Any attempt via the tcp-dump/wireshark is not acceptable by the certification. So it needs to be only the syslogs. I think from 9.0.65 it should be easy. For the existing versions yes the log needs to be in syslog until it rotates. If it gives cipher details that’s good, but importantly it should give the Ips. Once again thanks a lot for your overwhelming responses. If I will be able to close this today, it is pretty great. Also let me know in 9.0.65 is there any detailed attempt made to log about the ssl handshake including the ciphers etc.,? Regards, Raghav From: Christopher Schultz Date: Friday, 8 July 2022 at 12:05 AM To: users@tomcat.apache.org Subject: Re: AW: SSL handshake failure logs required for auditing purpose Thomas, On 7/7/22 13:36, Thomas Hoffmann (Speed4Trade GmbH) wrote: > > >> -Ursprüngliche Nachricht- >> Von: Thomas Hoffmann (Speed4Trade GmbH) >> >> Gesendet: Donnerstag, 7. Juli 2022 19:23 >> An: Tomcat Users List >> Betreff: AW: SSL handshake failure logs required for auditing purpose >> >> Hello Raghav, >> >>> -Ursprüngliche Nachricht- >>> Von: Ragavendhiran Bhiman (rabhiman) >>> Gesendet: Donnerstag, 7. Juli 2022 18:13 >>> An: Tomcat Users List >>> Betreff: Re: SSL handshake failure logs required for auditing purpose >>> >>> Version of tomcat used 9.0.x. >>> Kindly help on the ssl logging for auditing purpose other than -D >>> javax.net option. >>> >>> From: Ragavendhiran Bhiman (rabhiman) >>> Date: Thursday, 7 July 2022 at 9:41 PM >>> To: users@tomcat.apache.org >>> Subject: SSL handshake failure logs required for auditing purpose Hi >>> All, >>> >>> I require your kind help in logging the SSl connection failure logs >>> including iP in the tomcat, Is there any best way to do It without >>> performance impact other than -Djava.net debugs in jdk, is there any >>> direct way from tomcat? Or any way we can derive any class from JSSE >>> extension classes and add HandShakeListener while using the >>> connectors. All our SSL connections are going through connectors. So >>> kindly need your help how to log those SSL connection auditing logs >> through best method. >>> Thanks a lot in advance. >>> >>> Regards, >>> Raghav >> >> Which OS are you using? >> Can you use Wireshark or TCPDump for your purposes? >> If you are using Chrome or FF as Client, you can set the environment variable >> SSLKEYLOGFILE to write the current key to a file which Wireshark can take to >> decrypt the traffic. >> >> The handshake itself is not encrypted. If the handshake is enough, TCPDump >> or Wireshark are sufficient. >> >> Greetings, >> Thomas >> > > Short Addendum: > 1) Do you want to write the log permanently or just for an audit session? > 2) Which details do you want to log? Agreed cipher? Offered ciphers by the > client? SNI-header? ...? > 3) What is the purpose of the logging? > Insecure ciphers can be mitigated by server configuration. I think he wants to implement a poor-mans NIDS. Raghav, please be aware that any web browser that first attempts to use a SSLv3/TLSv1/TLSv1.3 handshake, fails, and retries with a TLSv1.2/similar handshake will cause massive numbers of false-positives in your logs. I would ask whoever is requesting this logging why they are looking at such failures. Handshake failures are not always indicative of some kind of intrusion attempt. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SSL handshake failure logs required for auditing purpose
Version of tomcat used 9.0.x. Kindly help on the ssl logging for auditing purpose other than -D javax.net option. From: Ragavendhiran Bhiman (rabhiman) Date: Thursday, 7 July 2022 at 9:41 PM To: users@tomcat.apache.org Subject: SSL handshake failure logs required for auditing purpose Hi All, I require your kind help in logging the SSl connection failure logs including iP in the tomcat, Is there any best way to do It without performance impact other than -Djava.net debugs in jdk, is there any direct way from tomcat? Or any way we can derive any class from JSSE extension classes and add HandShakeListener while using the connectors. All our SSL connections are going through connectors. So kindly need your help how to log those SSL connection auditing logs through best method. Thanks a lot in advance. Regards, Raghav
SSL handshake failure logs required for auditing purpose
Hi All, I require your kind help in logging the SSl connection failure logs including iP in the tomcat, Is there any best way to do It without performance impact other than -Djava.net debugs in jdk, is there any direct way from tomcat? Or any way we can derive any class from JSSE extension classes and add HandShakeListener while using the connectors. All our SSL connections are going through connectors. So kindly need your help how to log those SSL connection auditing logs through best method. Thanks a lot in advance. Regards, Raghav
Re: Need help on ssl handshake logging for audit purpose
Any help please? From: Ragavendhiran Bhiman (rabhiman) Date: Wednesday, 9 June 2021 at 7:37 PM To: Tomcat Users List Subject: Re: Need help on ssl handshake logging for audit purpose Hi John I am getting the output like this 10966181161114832473721710433823523866273491920411012289522541835156 0451112281652151321572412101201618710026238431618119549401121401171233324158 42501341314694224184221573623625026123232102239748963162282031315479568415410816714824988456515811161712381042357237101163207528011232191246200181691271811772110: A8 3C 2E B6 4E DD 31 26 AB F1 06 00 8C 2E E8 4F .<..N.1&...O 0010: D4 92 83 35 E1 DA 1D E7 A1 C8 CB E5 5A 2E DB FA ...5Z... : 60 C0 4A 1A 6D 42 B5 A1 72 53 F7 25 D9 68 03 26 `.J.mB..rS.%.h.& 0010: EB EE 42 1B 03 31 13 CC 6E 7A 59 34 FE B7 05 9C ..B..1..nzY4 : 22 70 51 FA 2D 6F E4 A5 D7 84 9D F1 D2 78 A1 57 "pQ.-o...x.W 0010: 64 1A EE 2B A1 51 C3 31 28 70 8C 75 7B 21 F1 3A d..+.Q.1(p.u.!.: : 83 04 6C 63 69 8A E8 0B F5 43 9E 05 0C 0A E8 16 ..lciC.. 0010: 96 93 E4 6A 87 13 F6 7D 0D 1D 20 08 27 31 82 27 ...j.. .'1.' 0020: 1B 8A F0 18 D2 DD 5E BB 0A DF A6 E2 14 81 E3 DC ..^. : 4A D4 E5 68 C2 63 F8 91 3D 76 B3 41 41 07 67 6D J..h.c..=v.AA.gm : 57 8A 27 73 2F B5 16 24 03 A3 21 37 D1 F4 29 32 W.'s/..$..!7..)2 : 4C B5 BB 4FL..O : B4 B7 59 3B..Y; 22618924724621010316224313810322810210322025425412734108751091578720586134212421861742242322282151444201162195251381361052262242071952223010517584 086134212421861742242322282151444201162195251381361052262242071952223010517584 050168210587015618110341164162418918160185202512541454715622811317821717232 050168210587015618110341164162418918160185202512541454715622811317821717232 017018119731266076432682421698238201241421872101374251165943619924771 017018119731266076432682421698238201241421872101374251165943619924771 0197513234491791511232391311646424511639144164531851431382451535291109105173 0197513234491791511232391311646424511639144164531851431382451535291109105173 0165273404544562229869431866751661420731628519075881212516497 0165273404544562229869431866751661420731628519075881212516497 0162215220131114313012964121735315431197818226189183158121672152730255135 0162215220131114313012964121735315431197818226189183158121672152730255135 0174251623522412132441636156193199643314391140113617521632383522891177 0174251623522412132441636156193199643314391140113617521632383522891177 016172153323923311721332236451412341979725524824323115820018920327155244 016172153323923311721332236451412341979725524824323115820018920327155244 02555162931826317721088252116103172112174681781811831591951062001732151285667 02555162931826317721088252116103172112174681781811831591951062001732151285667 02434513417921281596377905722819048352231781071546621230722431053615071 02434513417921281596377905722819048352231781071546621230722431053615071 0341744557183234171232141168229125938115013199142137050247921521265183188 0341744557183234171232141168229125938115013199142137050247921521265183188 013962213818324423661197416712420868203102001795024773201441213623025467 013962213818324423661197416712420868203102001795024773201441213623025467 02263295187184129931764824618134152429721124328179572171522446620714230247 02263295187184129931764824618134152429721124328179572171522446620714230247 025210871552362171396122248195434021117619979659286210176194770163241139 025210871552362171396122248195434021117619979659286210176194770163241139 028159162521301292351143532172061711341592073874203241142251185189489420 028159162521301292351143532172061711341592073874203241142251185189489420 03956222114828171171751315334156124240187685701262397019722317714025541 020363141169591622613713314615297431001152129929143100621753222047191159 : 60 C0 4A 36 03 5F 3E DD 94 1C AB AB 4B 83 35 22 `.J6._>.K.5" 0010: 9C 7C F0 BB 44 39 00 7E EF 46 C5 DF B1 8C FF 29 D9...F.) : 83 F1 8F 38 CB 3F 8D A9 3B A2 1A 89 85 92 98 61 ...8.?..;..a 0010: 2B 64 73 D4 63 1D 8F 64 3E 11 35 16 CC 47 5B 9F +ds.c..d>.5..G[. : FC 59 08 24 49 7F 24 E5 17 98 15 9C 6D 4D CF 34 .Y.$I.$.mM.4 0010: 45 C6 58 A9 45 5A 78 38 0D 4F 52 1A 65 43 B0 2C E.X.EZx8.OR.eC., 0020: 17 C4 B5 23 DE 57 15 60 2C 65 24 B6 2B 9A 31 AE ...#.W.`,e$.+.1. : CD F2 EE 3C 1C D1 1F 13 A5 EB AA 8A DA EF A2 09 ...< : 6F 30 03 F9 0E C4 79 F1 C8 2E E3 57 6F 90 26 C8 o0yWo.&. : 34 24 0E D74$.. : 2B D4 B7 01+... 18221835149659103228381331551575241147331031321291661321159 No client Hello messages etc., in the Catalina.out log. I tried to change the encoding but still the same issue. Thanks & Regards, Raghav From: john.e.gr...@wellsfargo.com.INVALID Date: Wedne
Re: Need help on ssl handshake logging for audit purpose
Hi John I am getting the output like this 10966181161114832473721710433823523866273491920411012289522541835156 0451112281652151321572412101201618710026238431618119549401121401171233324158 42501341314694224184221573623625026123232102239748963162282031315479568415410816714824988456515811161712381042357237101163207528011232191246200181691271811772110: A8 3C 2E B6 4E DD 31 26 AB F1 06 00 8C 2E E8 4F .<..N.1&...O 0010: D4 92 83 35 E1 DA 1D E7 A1 C8 CB E5 5A 2E DB FA ...5Z... : 60 C0 4A 1A 6D 42 B5 A1 72 53 F7 25 D9 68 03 26 `.J.mB..rS.%.h.& 0010: EB EE 42 1B 03 31 13 CC 6E 7A 59 34 FE B7 05 9C ..B..1..nzY4 : 22 70 51 FA 2D 6F E4 A5 D7 84 9D F1 D2 78 A1 57 "pQ.-o...x.W 0010: 64 1A EE 2B A1 51 C3 31 28 70 8C 75 7B 21 F1 3A d..+.Q.1(p.u.!.: : 83 04 6C 63 69 8A E8 0B F5 43 9E 05 0C 0A E8 16 ..lciC.. 0010: 96 93 E4 6A 87 13 F6 7D 0D 1D 20 08 27 31 82 27 ...j.. .'1.' 0020: 1B 8A F0 18 D2 DD 5E BB 0A DF A6 E2 14 81 E3 DC ..^. : 4A D4 E5 68 C2 63 F8 91 3D 76 B3 41 41 07 67 6D J..h.c..=v.AA.gm : 57 8A 27 73 2F B5 16 24 03 A3 21 37 D1 F4 29 32 W.'s/..$..!7..)2 : 4C B5 BB 4FL..O : B4 B7 59 3B..Y; 22618924724621010316224313810322810210322025425412734108751091578720586134212421861742242322282151444201162195251381361052262242071952223010517584 086134212421861742242322282151444201162195251381361052262242071952223010517584 050168210587015618110341164162418918160185202512541454715622811317821717232 050168210587015618110341164162418918160185202512541454715622811317821717232 017018119731266076432682421698238201241421872101374251165943619924771 017018119731266076432682421698238201241421872101374251165943619924771 0197513234491791511232391311646424511639144164531851431382451535291109105173 0197513234491791511232391311646424511639144164531851431382451535291109105173 0165273404544562229869431866751661420731628519075881212516497 0165273404544562229869431866751661420731628519075881212516497 0162215220131114313012964121735315431197818226189183158121672152730255135 0162215220131114313012964121735315431197818226189183158121672152730255135 0174251623522412132441636156193199643314391140113617521632383522891177 0174251623522412132441636156193199643314391140113617521632383522891177 016172153323923311721332236451412341979725524824323115820018920327155244 016172153323923311721332236451412341979725524824323115820018920327155244 02555162931826317721088252116103172112174681781811831591951062001732151285667 02555162931826317721088252116103172112174681781811831591951062001732151285667 02434513417921281596377905722819048352231781071546621230722431053615071 02434513417921281596377905722819048352231781071546621230722431053615071 0341744557183234171232141168229125938115013199142137050247921521265183188 0341744557183234171232141168229125938115013199142137050247921521265183188 013962213818324423661197416712420868203102001795024773201441213623025467 013962213818324423661197416712420868203102001795024773201441213623025467 02263295187184129931764824618134152429721124328179572171522446620714230247 02263295187184129931764824618134152429721124328179572171522446620714230247 025210871552362171396122248195434021117619979659286210176194770163241139 025210871552362171396122248195434021117619979659286210176194770163241139 028159162521301292351143532172061711341592073874203241142251185189489420 028159162521301292351143532172061711341592073874203241142251185189489420 03956222114828171171751315334156124240187685701262397019722317714025541 020363141169591622613713314615297431001152129929143100621753222047191159 : 60 C0 4A 36 03 5F 3E DD 94 1C AB AB 4B 83 35 22 `.J6._>.K.5" 0010: 9C 7C F0 BB 44 39 00 7E EF 46 C5 DF B1 8C FF 29 D9...F.) : 83 F1 8F 38 CB 3F 8D A9 3B A2 1A 89 85 92 98 61 ...8.?..;..a 0010: 2B 64 73 D4 63 1D 8F 64 3E 11 35 16 CC 47 5B 9F +ds.c..d>.5..G[. : FC 59 08 24 49 7F 24 E5 17 98 15 9C 6D 4D CF 34 .Y.$I.$.mM.4 0010: 45 C6 58 A9 45 5A 78 38 0D 4F 52 1A 65 43 B0 2C E.X.EZx8.OR.eC., 0020: 17 C4 B5 23 DE 57 15 60 2C 65 24 B6 2B 9A 31 AE ...#.W.`,e$.+.1. : CD F2 EE 3C 1C D1 1F 13 A5 EB AA 8A DA EF A2 09 ...< : 6F 30 03 F9 0E C4 79 F1 C8 2E E3 57 6F 90 26 C8 o0yWo.&. : 34 24 0E D74$.. : 2B D4 B7 01+... 18221835149659103228381331551575241147331031321291661321159 No client Hello messages etc., in the Catalina.out log. I tried to change the encoding but still the same issue. Thanks & Regards, Raghav From: john.e.gr...@wellsfargo.com.INVALID Date: Wednesday, 9 June 2021 at 7:20 PM To: users@tomcat.apache.org Subject: RE: Need help on ssl handshake logging for audit purpose Raghav, > -Original Message----- > From: Ragavendhira
Re: Need help on ssl handshake logging for audit purpose
Kindly help me on the below. Thanks a lot for the help. From: Ragavendhiran Bhiman (rabhiman) Date: Tuesday, 8 June 2021 at 7:18 PM To: users@tomcat.apache.org Subject: Need help on ssl handshake logging for audit purpose Hi All, In our product we are using jdk8 and tomcat apache latest version. I have enabled -Djavax.net.debug=ssl:handshake from jdk side. But I could see the handshake logging are coming as hex in the Catalina.out log messages. I want to know how to print the message in the proper English format. Is any other mistake I am doing? Kindly help me in this regard. Thanks & Regards, Raghav
Need help on ssl handshake logging for audit purpose
Hi All, In our product we are using jdk8 and tomcat apache latest version. I have enabled -Djavax.net.debug=ssl:handshake from jdk side. But I could see the handshake logging are coming as hex in the Catalina.out log messages. I want to know how to print the message in the proper English format. Is any other mistake I am doing? Kindly help me in this regard. Thanks & Regards, Raghav
Re: Memory leak in the PKCS11 how to fix the problem
The tomcat version is apache-tomcat-8.5.29
And RedHat Enterprise Linux 7.6
On 22/07/20, 7:28 PM, "Ragavendhiran Bhiman (rabhiman)"
wrote:
Hi Chris,
Please see my answers inline.
Also shared the svg graph here.
https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0
Kindly reply.
Thanks & Regards,
Raghav
On 20/07/20, 11:08 PM, "Christopher Schultz"
wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Ragavendhiran,
On 7/18/20 08:53, Ragavendhiran Bhiman (rabhiman) wrote:
> The OS is RHEL => 7.6
>
>
> From: "Ragavendhiran Bhiman (rabhiman)" Date:
> Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List
> Subject: Memory leak in the PKCS11 how to
> fix the problem
>
>
>
> From: "Ragavendhiran Bhiman (rabhiman)" Date:
> Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List
> Subject: Memory leak in the PKCS11 how to
> fix the problem
>
> Hello All,
>
> I am seeing the memory leaks from tomcat apache in the following
> SSL path using PKCS11.
In what way are you using PKCS#11? Using a hardware crypto engine via
OpenSSL? Are you using the APR connector or are you using OpenSSL via
JSSE?
It is being used via the OpenSSL. I am not sure how to identify
it is via the APR or hardware.
> Attached the flame graph of memory possible memory leaks in this
> area. Please check the attached flame graph of the memory trace.
Attachments are stripped from messages to this list. Can you post the
graph somewhere and include a link to a reply here?
Attached again using the dropbox.
> On simply a long run the memory keep on allocated in these back
> traces only causing the memory leak, and the polling of the async
> profiler for more than 6hours shows this clearly. Could you please
> help how to fix this problem? (open this svg graph in browser
> only)> Note: If C_DestroyObject is not called because of finalizer
> accumulation is also tested by inducing the gc using the jmap
> command still could see the memory never gone down after the Full
> GC collection as well. Expecting your advice on the same.
Are you suggesting that the solution will be to do something explicit
in a class finalizer? Or are you suggesting that some additional
native calls are required to clean-up.
I called the GC via jmap -histo:live but still the weak references
not cleaned up the RSS never reduced. This means that some where the memory
being held in native that’s what my suspect. Kindly suggest from your side as
well.
Thanks.
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8V1jYACgkQHPApP6U8
pFj3Rw/7B9SxlYNFPZoM6O4GWnwlxI2u+tLrsk2x/6TNOQuFnOIACctTKVe/WIwI
/kF3O7kqwfDEewDhNa6KndUpQgWcvrtbON/QXEAK8SE7HaP0uZv0n5fB6FyRNPPo
uJpnVu5WkJuvFcc9VmluMk9y+SdijhucQtIiK8Y+JEeHmp3Jjg3b45TeJYupDG3K
iAuKqjgWHdRwpVgUNIKxj27PJJBs+qG8yfg7qkVvHrJHzNvZv5rGImcjtwPO6ktQ
U7yWIcJLldF4pkOAWGSiqaCbZsFUCVRK8euejAj5kThZRELKtrJq+QOCy+pIiYHe
rvJpJyG44VLYMqYDMPkDrHVrSJIj88zSJNdfBWXvnC2Ol7dJRTfpau19Js/kYwls
53cELUg7XNOLpH14KnAKk2XMh25xjuZTHLtj2G3p61Frb48+81Ns6X8Hh/oDt/n+
vGUpwU1Crf2HuXg1YiF4L3aw6UnnS3vH/BeeZpj6z8qwFdw9O7ylAcXOndhAiyN0
SxQYmJIXkL8+WXBSTWt7dWZBdYiml/Zl2lkDhRGh+ftCCYZ8UgK5d2+b2HHVQiot
nEs/dAu8rESBkFe0zXB9fgSeviLNwBDkAvenFJ/kZMstT5tzFZiLZW4EKpB8OWTx
fbOGK5KQZB37Cbv+Eq64axWEK3vQHYROfeeAzEz3NgmYE2w74Rg=
=yShM
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
?B�CB�?�?[��X��ܚX�K??K[XZ[?�?\�\��][��X��ܚX�P??�X�]?�\?X�?K�ܙ�B��܈?Y??]?[ۘ[??��[X[�?�??K[XZ[?�?\�\��Z?[�X�]?�\?X�?K�ܙ�B�
Re: Memory leak in the PKCS11 how to fix the problem
Hi Chris,
Please see my answers inline.
Also shared the svg graph here.
https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0
Kindly reply.
Thanks & Regards,
Raghav
On 20/07/20, 11:08 PM, "Christopher Schultz"
wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Ragavendhiran,
On 7/18/20 08:53, Ragavendhiran Bhiman (rabhiman) wrote:
> The OS is RHEL => 7.6
>
>
> From: "Ragavendhiran Bhiman (rabhiman)" Date:
> Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List
> Subject: Memory leak in the PKCS11 how to
> fix the problem
>
>
>
> From: "Ragavendhiran Bhiman (rabhiman)" Date:
> Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List
> Subject: Memory leak in the PKCS11 how to
> fix the problem
>
> Hello All,
>
> I am seeing the memory leaks from tomcat apache in the following
> SSL path using PKCS11.
In what way are you using PKCS#11? Using a hardware crypto engine via
OpenSSL? Are you using the APR connector or are you using OpenSSL via
JSSE?
It is being used via the OpenSSL. I am not sure how to identify it
is via the APR or hardware.
> Attached the flame graph of memory possible memory leaks in this
> area. Please check the attached flame graph of the memory trace.
Attachments are stripped from messages to this list. Can you post the
graph somewhere and include a link to a reply here?
Attached again using the dropbox.
> On simply a long run the memory keep on allocated in these back
> traces only causing the memory leak, and the polling of the async
> profiler for more than 6hours shows this clearly. Could you please
> help how to fix this problem? (open this svg graph in browser
> only)> Note: If C_DestroyObject is not called because of finalizer
> accumulation is also tested by inducing the gc using the jmap
> command still could see the memory never gone down after the Full
> GC collection as well. Expecting your advice on the same.
Are you suggesting that the solution will be to do something explicit
in a class finalizer? Or are you suggesting that some additional
native calls are required to clean-up.
I called the GC via jmap -histo:live but still the weak references not
cleaned up the RSS never reduced. This means that some where the memory being
held in native that’s what my suspect. Kindly suggest from your side as well.
Thanks.
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8V1jYACgkQHPApP6U8
pFj3Rw/7B9SxlYNFPZoM6O4GWnwlxI2u+tLrsk2x/6TNOQuFnOIACctTKVe/WIwI
/kF3O7kqwfDEewDhNa6KndUpQgWcvrtbON/QXEAK8SE7HaP0uZv0n5fB6FyRNPPo
uJpnVu5WkJuvFcc9VmluMk9y+SdijhucQtIiK8Y+JEeHmp3Jjg3b45TeJYupDG3K
iAuKqjgWHdRwpVgUNIKxj27PJJBs+qG8yfg7qkVvHrJHzNvZv5rGImcjtwPO6ktQ
U7yWIcJLldF4pkOAWGSiqaCbZsFUCVRK8euejAj5kThZRELKtrJq+QOCy+pIiYHe
rvJpJyG44VLYMqYDMPkDrHVrSJIj88zSJNdfBWXvnC2Ol7dJRTfpau19Js/kYwls
53cELUg7XNOLpH14KnAKk2XMh25xjuZTHLtj2G3p61Frb48+81Ns6X8Hh/oDt/n+
vGUpwU1Crf2HuXg1YiF4L3aw6UnnS3vH/BeeZpj6z8qwFdw9O7ylAcXOndhAiyN0
SxQYmJIXkL8+WXBSTWt7dWZBdYiml/Zl2lkDhRGh+ftCCYZ8UgK5d2+b2HHVQiot
nEs/dAu8rESBkFe0zXB9fgSeviLNwBDkAvenFJ/kZMstT5tzFZiLZW4EKpB8OWTx
fbOGK5KQZB37Cbv+Eq64axWEK3vQHYROfeeAzEz3NgmYE2w74Rg=
=yShM
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Memory leak in the PKCS11 how to fix the problem
Sorry Martin, My answers are inline. Thanks & Regards, Raghav On 22/07/20, 7:20 PM, "Ragavendhiran Bhiman (rabhiman)" wrote: Hello Martin, Thanks for your reply https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0 Here is the link I have shared the flame graph. Also kindly check my answers in red as well. Thanks & Regards, Raghav On 22/07/20, 2:33 PM, "Martin Grigorov" wrote: Hi Ragavendhiran, On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman) wrote: > Hello All, > > > > I am seeing the memory leaks from tomcat apache in the following SSL path > using PKCS11. Attached the flame graph of memory possible memory leaks in > this area. > > Please check the attached flame graph of the memory trace. On simply a > long run the memory keep on allocated in these back traces only causing the > memory leak, and the polling of the async profiler for more than 6hours > shows this clearly. Could you please help how to fix this problem? > > (open this svg graph in browser only) > > > > Note: If C_DestroyObject is not called because of finalizer accumulation > is also tested by inducing the gc using the jmap command still could see > the memory never gone down after the Full GC collection as well. Expecting > your advice on the same. > With AsyncProfiler '-e alloc' you can see what part of the code is responsible for making most of the memory allocations, but it doesn't tell you whether those objects leak or not. AsyncProfiler helps you to identify the top allocations and if you manage to reduce them then you will reduce the GC runs and the time they take. To debug memory leaks you need to take a heap dump, e.g. with 'jmap -dump:live,format=b,file=heap.bin ' and analyze it. I'd recommend you to use Eclipse MAT to do that. "Yes with async profiler and just leaving the server without any sequence of action the memory started growing (RSS grows) when I profiled during that sequence there is no other memory allocation happening except this one. That’s why I am suspecting this flow clearly. Samples are only through this flow only". Also in your flame graph I see that Netty is responsible for 49.04% of the allocated objects and Tomcat for just 25.32%. > > Regards, > > Raghav > > Infrastructure engineer, > > Cisco ISE. > > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Memory leak in the PKCS11 how to fix the problem
Hello Martin, Thanks for your reply https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0 Here is the link I have shared the flame graph. Also kindly check my answers in red as well. Thanks & Regards, Raghav On 22/07/20, 2:33 PM, "Martin Grigorov" wrote: Hi Ragavendhiran, On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman) wrote: > Hello All, > > > > I am seeing the memory leaks from tomcat apache in the following SSL path > using PKCS11. Attached the flame graph of memory possible memory leaks in > this area. > > Please check the attached flame graph of the memory trace. On simply a > long run the memory keep on allocated in these back traces only causing the > memory leak, and the polling of the async profiler for more than 6hours > shows this clearly. Could you please help how to fix this problem? > > (open this svg graph in browser only) > > > > Note: If C_DestroyObject is not called because of finalizer accumulation > is also tested by inducing the gc using the jmap command still could see > the memory never gone down after the Full GC collection as well. Expecting > your advice on the same. > With AsyncProfiler '-e alloc' you can see what part of the code is responsible for making most of the memory allocations, but it doesn't tell you whether those objects leak or not. AsyncProfiler helps you to identify the top allocations and if you manage to reduce them then you will reduce the GC runs and the time they take. To debug memory leaks you need to take a heap dump, e.g. with 'jmap -dump:live,format=b,file=heap.bin ' and analyze it. I'd recommend you to use Eclipse MAT to do that. "Yes with async profiler and just leaving the server without any sequence of action the memory started growing (RSS grows) when I profiled during that sequence there is no other memory allocation happening except this one. That’s why I am suspecting this flow clearly. Samples are only through this flow only". Also in your flame graph I see that Netty is responsible for 49.04% of the allocated objects and Tomcat for just 25.32%. > > Regards, > > Raghav > > Infrastructure engineer, > > Cisco ISE. > > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Memory leak in the PKCS11 how to fix the problem
HI, Kindly suggest what could be the issue below. Thanks & Regards, Raghav From: "Ragavendhiran Bhiman (rabhiman)" Date: Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List Subject: Memory leak in the PKCS11 how to fix the problem From: "Ragavendhiran Bhiman (rabhiman)" Date: Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List Subject: Memory leak in the PKCS11 how to fix the problem Hello All, I am seeing the memory leaks from tomcat apache in the following SSL path using PKCS11. Attached the flame graph of memory possible memory leaks in this area. Please check the attached flame graph of the memory trace. On simply a long run the memory keep on allocated in these back traces only causing the memory leak, and the polling of the async profiler for more than 6hours shows this clearly. Could you please help how to fix this problem? (open this svg graph in browser only) Note: If C_DestroyObject is not called because of finalizer accumulation is also tested by inducing the gc using the jmap command still could see the memory never gone down after the Full GC collection as well. Expecting your advice on the same. Regards, Raghav Infrastructure engineer, Cisco ISE. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Memory leak in the PKCS11 how to fix the problem
Kindly reply for the below mail, Expecting your feedback so that I can take it forward. Thanks & Regards, Raghav On 18/07/20, 6:25 PM, "Ragavendhiran Bhiman (rabhiman)" wrote: The OS is RHEL => 7.6 From: "Ragavendhiran Bhiman (rabhiman)" Date: Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List Subject: Memory leak in the PKCS11 how to fix the problem From: "Ragavendhiran Bhiman (rabhiman)" Date: Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List Subject: Memory leak in the PKCS11 how to fix the problem Hello All, I am seeing the memory leaks from tomcat apache in the following SSL path using PKCS11. Attached the flame graph of memory possible memory leaks in this area. Please check the attached flame graph of the memory trace. On simply a long run the memory keep on allocated in these back traces only causing the memory leak, and the polling of the async profiler for more than 6hours shows this clearly. Could you please help how to fix this problem? (open this svg graph in browser only) Note: If C_DestroyObject is not called because of finalizer accumulation is also tested by inducing the gc using the jmap command still could see the memory never gone down after the Full GC collection as well. Expecting your advice on the same. Regards, Raghav Infrastructure engineer, Cisco ISE.
Re: Memory leak in the PKCS11 how to fix the problem
The OS is RHEL => 7.6 From: "Ragavendhiran Bhiman (rabhiman)" Date: Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List Subject: Memory leak in the PKCS11 how to fix the problem From: "Ragavendhiran Bhiman (rabhiman)" Date: Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List Subject: Memory leak in the PKCS11 how to fix the problem Hello All, I am seeing the memory leaks from tomcat apache in the following SSL path using PKCS11. Attached the flame graph of memory possible memory leaks in this area. Please check the attached flame graph of the memory trace. On simply a long run the memory keep on allocated in these back traces only causing the memory leak, and the polling of the async profiler for more than 6hours shows this clearly. Could you please help how to fix this problem? (open this svg graph in browser only) Note: If C_DestroyObject is not called because of finalizer accumulation is also tested by inducing the gc using the jmap command still could see the memory never gone down after the Full GC collection as well. Expecting your advice on the same. Regards, Raghav Infrastructure engineer, Cisco ISE.
Re: suspected memory leak in io.netty.buffer.PoolChunk
Hi All, When I analysed the heap dump the primary suspect shows that 54.21% occupied by the from io.netty.buffer.PoolChunk memory leak. Kindly help me to proceed further to solve the problem. Apache used version is 8.5.29. Thanks & Regards, Raghav On 29/04/20, 6:53 PM, "Ragavendhiran Bhiman (rabhiman)" wrote: This question is from 8.5.29 version. On 29/04/20, 6:52 PM, "Ragavendhiran Bhiman (rabhiman)" wrote: Hi, I am seeing a continuous memory leak from io.netty.buffer.PoolChunk, Kindly advice how to go ahead and trace the problem? Thanks a lot. Regards, Raghavendran +91 8220757651.
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Hello Chris,
Please see my reply below in line.
Thanks & Regards,
Raghav
On 30/04/20, 9:23 AM, "Christopher Schultz"
wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Raghav,
On 4/29/20 22:26, Ragavendhiran Bhiman (rabhiman) wrote:
> The below is the executer element.
>
> maxThreads="450" minSpareThreads="5"/>
>
> I also captured the network pcap and able to see many RST packets
> in between that is marked as RED in wireshark.
Okay, so far you have told us:
1. You are using Tomcat 8.5.29
[Raghav] yes correct.
2. You have an with 450 threads in it
[Raghav] Executor has 450 threads.
3. You see "lots of threads"
[Raghav] Yes above 450 threads or all the. 450 threads are alive.
4. You are seeing lots of RST packets.
[Raghav]I can provide one snapshot of pcap analyisis after the application data
exchange only I am able to see the RST packet which I see could be ok or sent
for socket closing.
We can't help you without more details. Pretend we aren't looking at
your screen as you investigate.
How many threads are you seeing? More than 450? How many, exactly?
What are the names of the threads?
[Raghav] Yes all the 450 threads are alive.
Some things I have noticed that seem ... suspicious.
1. Your sslImplementationName is invalid.
[Raghav] is the SSL implementation name has to be changed? What is the SSL name
to be used?
2. You have a 5-minute keepAliveTimeout -- which sounds insanely high
- -- and an infinite number of keepalive requests. Are you fronting
Tomcat with a load-balancer or other reverse-proxy?
[Raghav]. Yes, 5 minutes timeout is high. I already informed to the team. But
people are saying previous versions are working properly where the large number
of threads are not there.
I think the threads are based on the number of requests is it true?
3. You have sendReasonPhrase="true" which indicates that you are
working with clients which violate the HTTP specification.
[Raghav] this doesn't matter. I don’t know how this has been set to true.
4. You have an infinite "maxSavePostSize" setting. Are you expecting
many users to perform unauthenticated POSTs where the POST body needs
to be very large, and saved-and-replayed during the authentication step?
[Raghav] This is not known.
5. Your keystoreType is PKCS11 which is usually a hardware keystore.
Fine. But you have a truststoreType of PKCS11 as well. Are you using a
hardware trust store as well?
[Raghav] Not sure about this. There are many ciphers defined along with this as
well.
- -chris
>
> Thanks & Regards,
>
> Raghav
>
>
> On 29/04/20, 9:52 PM, "Mark Thomas" wrote:
>
> On 29/04/2020 14:53, Ragavendhiran Bhiman (rabhiman) wrote:
>> Yes you are correct apache tomcat version 8.5.29 being used.
>>
>> On 29/04/20, 7:22 PM, "Ragavendhiran Bhiman (rabhiman)"
>> wrote:
>>
>> Hi Mark,
>>
>> We have configured 450 threads for port number 443 with the
>> following executer
>
> That is a Connector element, not the executor element. We need
> both.
>
> Mark
>
>
>>
>> > protocol="org.apache.coyote.http11.Http11NioProtocol"
>> SSLEnabled="true" maxKeepAliveRequests="-1"
>> keepAliveTimeout="30" executor="AdminExecutorPool"
>> maxSavePostSize="-1" scheme="https" secure="true"
>> enableLookups="false" disableUploadTimeout="true"
>> acceptCount="100" compression="on"
>> compressableMimeType="text/html,text/json,text/javascript,text/css,ap
plication/javascript"
>
>>
> sslEnabledProtocols="${sslEnabledProtocolsHighSecurity}" server="
> "
>> allowUnsafeLegacyRenegotiation="false" clientAuth="false"
>> bindOnInit="false" URIEncoding="UTF-8"
>> useBodyEncodingForURI="true" keystoreType="PKCS11"
>> keyAlias="tomcat" truststoreType="PKCS11"
>> sendReasonPhrase="true"
>> sslImplementationName="org.apache.tomcat.util.net.jsse.IseJSSEImpleme
ntation"
>
>>
>
>> />
>>
>> I could see 450 threads open for servicing the clients in one
>> specific setup only what could be the reason?
>>
>> Thanks a lot.
>>
>> Regards,
>>
>> Raghav
>>
>>
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Hi Mark,
The below is the executer element.
I also captured the network pcap and able to see many RST packets in between
that is marked as RED in wireshark.
Thanks & Regards,
Raghav
On 29/04/20, 9:52 PM, "Mark Thomas" wrote:
On 29/04/2020 14:53, Ragavendhiran Bhiman (rabhiman) wrote:
> Yes you are correct apache tomcat version 8.5.29 being used.
>
> On 29/04/20, 7:22 PM, "Ragavendhiran Bhiman (rabhiman)"
wrote:
>
> Hi Mark,
>
> We have configured 450 threads for port number 443 with the following
executer
That is a Connector element, not the executor element. We need both.
Mark
>
> port="443"
protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
> maxKeepAliveRequests="-1"
> keepAliveTimeout="30"
> executor="AdminExecutorPool"
> maxSavePostSize="-1"
> scheme="https" secure="true"
> enableLookups="false" disableUploadTimeout="true"
> acceptCount="100"
> compression="on"
>
compressableMimeType="text/html,text/json,text/javascript,text/css,application/javascript"
> sslEnabledProtocols="${sslEnabledProtocolsHighSecurity}"
server=" "
> allowUnsafeLegacyRenegotiation="false"
> clientAuth="false"
> bindOnInit="false"
> URIEncoding="UTF-8"
> useBodyEncodingForURI="true"
> keystoreType="PKCS11"
> keyAlias="tomcat"
> truststoreType="PKCS11"
> sendReasonPhrase="true"
>
sslImplementationName="org.apache.tomcat.util.net.jsse.IseJSSEImplementation"
>
> />
>
> I could see 450 threads open for servicing the clients in one
specific setup only what could be the reason?
>
> Thanks a lot.
>
> Regards,
>
> Raghav
>
>
> On 29/04/20, 7:18 PM, "Mark Thomas" wrote:
>
> On 29/04/2020 14:24, Ragavendhiran Bhiman (rabhiman) wrote:
> > Apache version 8.5.29
>
> Given this is the Apache Tomcat mailing list and that that is a
valid,
> although rather old, Tomcat version number I assume you mean you
are
> using Apache Tomcat 8.5.29.
>
> Generally, please also include JVM vendor and version being used
as well
> as OS.
>
>
>
> > Hi,
> >
> > I am seeing too many open threads to port number 443 with
TLSv1.2, what could be the primary reason for the same?
>
> Open threads? That doesn't make sense. Do you mean open ports,
threads
> (idle, active, both) or something else?
>
> How are you defining "too many"? More than you expect? There is an
> error? Something else?
>
> > How can I analyze the problem ? Any particular pointers if you
could provide will be more helpful.
>
> That depends on what the problem turns out to be.
>
> Mark
>
>
-
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Yes you are correct apache tomcat version 8.5.29 being used. On 29/04/20, 7:22 PM, "Ragavendhiran Bhiman (rabhiman)" wrote: Hi Mark, We have configured 450 threads for port number 443 with the following executer I could see 450 threads open for servicing the clients in one specific setup only what could be the reason? Thanks a lot. Regards, Raghav On 29/04/20, 7:18 PM, "Mark Thomas" wrote: On 29/04/2020 14:24, Ragavendhiran Bhiman (rabhiman) wrote: > Apache version 8.5.29 Given this is the Apache Tomcat mailing list and that that is a valid, although rather old, Tomcat version number I assume you mean you are using Apache Tomcat 8.5.29. Generally, please also include JVM vendor and version being used as well as OS. > Hi, > > I am seeing too many open threads to port number 443 with TLSv1.2, what could be the primary reason for the same? Open threads? That doesn't make sense. Do you mean open ports, threads (idle, active, both) or something else? How are you defining "too many"? More than you expect? There is an error? Something else? > How can I analyze the problem ? Any particular pointers if you could provide will be more helpful. That depends on what the problem turns out to be. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Hi Mark, We have configured 450 threads for port number 443 with the following executer I could see 450 threads open for servicing the clients in one specific setup only what could be the reason? Thanks a lot. Regards, Raghav On 29/04/20, 7:18 PM, "Mark Thomas" wrote: On 29/04/2020 14:24, Ragavendhiran Bhiman (rabhiman) wrote: > Apache version 8.5.29 Given this is the Apache Tomcat mailing list and that that is a valid, although rather old, Tomcat version number I assume you mean you are using Apache Tomcat 8.5.29. Generally, please also include JVM vendor and version being used as well as OS. > Hi, > > I am seeing too many open threads to port number 443 with TLSv1.2, what could be the primary reason for the same? Open threads? That doesn't make sense. Do you mean open ports, threads (idle, active, both) or something else? How are you defining "too many"? More than you expect? There is an error? Something else? > How can I analyze the problem ? Any particular pointers if you could provide will be more helpful. That depends on what the problem turns out to be. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Apache version 8.5.29 From: "Ragavendhiran Bhiman (rabhiman)" Date: Wednesday, 29 April 2020 at 6:50 PM To: "users-ow...@tomcat.apache.org" , "users@tomcat.apache.org" Subject: Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads Adding tomcat users as well. From: "Ragavendhiran Bhiman (rabhiman)" Date: Wednesday, 29 April 2020 at 6:45 PM To: "users-ow...@tomcat.apache.org" Subject: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads Hi, I am seeing too many open threads to port number 443 with TLSv1.2, what could be the primary reason for the same? How can I analyze the problem ? Any particular pointers if you could provide will be more helpful. Thanks & Regards, Raghavendran +91 8220757651.
Re: suspected memory leak in io.netty.buffer.PoolChunk
This question is from 8.5.29 version. On 29/04/20, 6:52 PM, "Ragavendhiran Bhiman (rabhiman)" wrote: Hi, I am seeing a continuous memory leak from io.netty.buffer.PoolChunk, Kindly advice how to go ahead and trace the problem? Thanks a lot. Regards, Raghavendran +91 8220757651.
suspected memory leak in io.netty.buffer.PoolChunk
Hi, I am seeing a continuous memory leak from io.netty.buffer.PoolChunk, Kindly advice how to go ahead and trace the problem? Thanks a lot. Regards, Raghavendran +91 8220757651.
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Adding tomcat users as well. From: "Ragavendhiran Bhiman (rabhiman)" Date: Wednesday, 29 April 2020 at 6:45 PM To: "users-ow...@tomcat.apache.org" Subject: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads Hi, I am seeing too many open threads to port number 443 with TLSv1.2, what could be the primary reason for the same? How can I analyze the problem ? Any particular pointers if you could provide will be more helpful. Thanks & Regards, Raghavendran +91 8220757651.
