Re: AW: Tomcat 9.0.76 Memory leak with Java 17
Thomas, On 7/13/23 02:19, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello, -Ursprüngliche Nachricht- Von: Christopher Schultz Gesendet: Mittwoch, 12. Juli 2023 21:34 An: users@tomcat.apache.org Betreff: Re: Tomcat 9.0.76 Memory leak with Java 17 Michael, On 7/12/23 07:33, Michael Osipov wrote: On 2023/07/11 18:16:24 Christopher Schultz wrote: You should report all of the previous issues to Oracle against their ORDS version 22.1 and ask them to fix them. It's why you write those big, fat checks in the first place ;) This doesn't really matter. I have reported a memory leak in OJDBC many years ago where a background thread pins the WebappClassLoader. Answer from Oracle: This happens only once, does not repeat. We won't address that... Fair enough, but that doesn't mean it's not worth reporting. BTW the fine folks at Sun/Oracle and Connector/J are similarly confused when it comes to ClassLoader pinning in /their/ JDBC driver. I think they had a few different patches that all waved their hands _still_ failing to actually understand and fix the problem. -chris It's not just Oracle JDBC driver. Same can happen with SQL-Server JDBC driver when you use Kerberos Auth. Ticket was also closed without a fix. Yeah, and I think it just comes down to failure to understand the problem. The problem with Connector/J has always been that they launch a background thread to clean things up. All they have to do is provide a method to "shut down background threads". I kept getting pushback like "they are daemon threads, they won't keep your JVM running when your application stops" and I was like "hey sometimes the application stops and restarts without shutting down the JVM #webapps" and they were like "what does that even mean CLOSED WONTFIX". -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: Tomcat 9.0.76 Memory leak with Java 17
Hello, > -Ursprüngliche Nachricht- > Von: Christopher Schultz > Gesendet: Mittwoch, 12. Juli 2023 21:34 > An: users@tomcat.apache.org > Betreff: Re: Tomcat 9.0.76 Memory leak with Java 17 > > Michael, > > On 7/12/23 07:33, Michael Osipov wrote: > > On 2023/07/11 18:16:24 Christopher Schultz wrote: > >> You should report all of the previous issues to Oracle against their > >> ORDS version 22.1 and ask them to fix them. It's why you write those > >> big, fat checks in the first place ;) > > > > This doesn't really matter. I have reported a memory leak in OJDBC > > many years ago where a background thread pins the WebappClassLoader. > > Answer from Oracle: This happens only once, does not repeat. We won't > > address that... > Fair enough, but that doesn't mean it's not worth reporting. > > BTW the fine folks at Sun/Oracle and Connector/J are similarly confused > when it comes to ClassLoader pinning in /their/ JDBC driver. I think they had > a > few different patches that all waved their hands _still_ failing to actually > understand and fix the problem. > > -chris > It's not just Oracle JDBC driver. Same can happen with SQL-Server JDBC driver when you use Kerberos Auth. Ticket was also closed without a fix.
RE: Tomcat 9.0.76 Memory leak with Java 17
Chris, Yes it is unintentional. Actually once we start it with the Windows service, and run through a few reports on the website, it stops in just ba few minutes. We will look at the java heap size settings. Regards, James Boggs -Original Message- From: Christopher Schultz Sent: Wednesday, July 12, 2023 3:55 PM To: users@tomcat.apache.org Subject: Re: Tomcat 9.0.76 Memory leak with Java 17 James, On 7/12/23 15:41, James Boggs wrote: > Thanks for the input. I will forward the email to our developers to > look at the heap size settings being different. > > We have a Windows service that is used to start/stop Tomcat. When this > happens we find that the Windows service is no longer running. Depending upon your settings, you may have more than one log file to look at. Look at everything in CATALINA_BASE/logs the next time the service goes down to see if there is a clue in any of them. You could also look for hs_pid[process-id] files (or the Windows equivalent, if it's different) that are usually dumped when a JVM crash occurs. Each of your logs seems to show that Tomcat is being shut-down in an orderly way, though. Are you sure it's crashing and not being intentionally stopped? -chris > -Original Message- > From: Christopher Schultz > Sent: Wednesday, July 12, 2023 3:32 PM > To: users@tomcat.apache.org > Subject: Re: Tomcat 9.0.76 Memory leak with Java 17 > > Suvendu, > > On 7/12/23 07:11, Suvendu Sekhar Mondal wrote: >> On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz >> wrote: >>> >>> James, >>> >>> On 7/11/23 10:21, James Boggs wrote: >>>> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on >>>> Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL. >>>> >>>> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 >>>> and to ORDS 22.1, then used Java 17 to create a new Java Keystore >>>> and a new SSL csr file, and imported a new SSL certificate from the >>>> CA into the new keystore. >>>> >>>> The website works but after logging in there are memory leak >>>> warnings and the Tomcat service crashes within just a couple of minutes. >>>> >>>> We even upgraded to 9.0.76 and the issue persists. >>>> >>>> Below is an excerpt from the stderr log file. >>>> >>>> I have been unable to find any recent threads on this, any help is >>>> appreciated. Is any other information needed? >>> >>> I think you have included all necessary information. I'm >>> chopping-out the irrrelevant bits: >>> >>>> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd] >>>> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to >>>> unregister it when the web application was stopped. To prevent a >>>> memory leak, the JDBC Driver has been forcibly unregistered. >>> >>> This is actually "okay" in that Tomcat has detected a global JDBC >>> driver registration performed by the application, and is fixing the >>> problem for you. The application, however, is making a mistake by >>> not de-registering that JDBC driver. Your options are to move the >>> driver library from your application into Tomcat's lib/ directory, >>> fix the application to de-register the driver when it shuts down, or just >>> ignore the warning. >>> But you should fix the application. >>> >>>> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] >>>> appears to have started a thread named >>>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] >>>> but has failed to stop it. This is very likely to create a memory leak. >>>> Stack trace of thread: >>> >>> There are multiple instances of this same message and THIS is your >>> problem. The problem is what the error message says: your >>> application has started a Thread and never stopped it. The "memory >>> leak" comes from the fact that the Thread has inherited the web >>> application's ClassLoader and the web application is being >>> re-loaded. When that happens, Tomcat discards the ClassLoader which >>> usually means the GC will clean up after it at some point in the >>> future. But that Thread is still running and will keep the ClassLoader in >>> memory, likely forever. >>> >>> You have a few options: >>> >>> 1. Fix the application. The application needs to shut-down any >>> threads is
Re: Tomcat 9.0.76 Memory leak with Java 17
James, On 7/12/23 15:41, James Boggs wrote: Thanks for the input. I will forward the email to our developers to look at the heap size settings being different. We have a Windows service that is used to start/stop Tomcat. When this happens we find that the Windows service is no longer running. Depending upon your settings, you may have more than one log file to look at. Look at everything in CATALINA_BASE/logs the next time the service goes down to see if there is a clue in any of them. You could also look for hs_pid[process-id] files (or the Windows equivalent, if it's different) that are usually dumped when a JVM crash occurs. Each of your logs seems to show that Tomcat is being shut-down in an orderly way, though. Are you sure it's crashing and not being intentionally stopped? -chris -Original Message- From: Christopher Schultz Sent: Wednesday, July 12, 2023 3:32 PM To: users@tomcat.apache.org Subject: Re: Tomcat 9.0.76 Memory leak with Java 17 Suvendu, On 7/12/23 07:11, Suvendu Sekhar Mondal wrote: On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz wrote: James, On 7/11/23 10:21, James Boggs wrote: We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL. We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL csr file, and imported a new SSL certificate from the CA into the new keystore. The website works but after logging in there are memory leak warnings and the Tomcat service crashes within just a couple of minutes. We even upgraded to 9.0.76 and the issue persists. Below is an excerpt from the stderr log file. I have been unable to find any recent threads on this, any help is appreciated. Is any other information needed? I think you have included all necessary information. I'm chopping-out the irrrelevant bits: 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. This is actually "okay" in that Tomcat has detected a global JDBC driver registration performed by the application, and is fixing the problem for you. The application, however, is making a mistake by not de-registering that JDBC driver. Your options are to move the driver library from your application into Tomcat's lib/ directory, fix the application to de-register the driver when it shuts down, or just ignore the warning. But you should fix the application. 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: There are multiple instances of this same message and THIS is your problem. The problem is what the error message says: your application has started a Thread and never stopped it. The "memory leak" comes from the fact that the Thread has inherited the web application's ClassLoader and the web application is being re-loaded. When that happens, Tomcat discards the ClassLoader which usually means the GC will clean up after it at some point in the future. But that Thread is still running and will keep the ClassLoader in memory, likely forever. You have a few options: 1. Fix the application. The application needs to shut-down any threads is starts during its operation, preferrably in a ServletContextListener's destroy method or similar. 2. Don't hot-reload your application. Instead, shut-down the JVM and re-start it. Ovbviously, this may have availability implications, but then again so does running out of memory and having to bounce the JVM, anyway. I was checking the stacks and saw this: They might not be doing "hot-deploy" of the app. AFAIK those messages come once someone stops Tomcat. 2023-07-10T20:52:06.292Z INFOStarting ProtocolHandler ["https-openssl-nio-10.2.251.132-443"] 2023-07-10T20:52:06.346Z INFOServer startup in [28980] milliseconds 2023-07-10T21:35:40.487Z INFOPausing ProtocolHandler ["https-openssl-nio-10.2.251.132-443"] 2023-07-10T21:35:40.495Z INFOStopping service [Catalina] 2023-07-10T21:35:40.910Z INFODestroyed pool : |default|lo|-2023-07-10T20-51-53.099285500Z Good point: Tomcat will complain about this stuff on shutdown even if the warnings are irrelevant because the application-stop leak-checks are run whether Tomcat is going to stop or not. So operationally, these warnings may be irrelevant. (IMHO you should still fix the application!) Another thing I noticed was the difference in allocated Java heap. It appears that first it sets min=max=1GB and after that it's setting
RE: Tomcat 9.0.76 Memory leak with Java 17
Thanks for the input. I will forward the email to our developers to look at the heap size settings being different. We have a Windows service that is used to start/stop Tomcat. When this happens we find that the Windows service is no longer running. Thanks, James Boggs -Original Message- From: Christopher Schultz Sent: Wednesday, July 12, 2023 3:32 PM To: users@tomcat.apache.org Subject: Re: Tomcat 9.0.76 Memory leak with Java 17 Suvendu, On 7/12/23 07:11, Suvendu Sekhar Mondal wrote: > On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz > wrote: >> >> James, >> >> On 7/11/23 10:21, James Boggs wrote: >>> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on >>> Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL. >>> >>> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and >>> to ORDS 22.1, then used Java 17 to create a new Java Keystore and a >>> new SSL csr file, and imported a new SSL certificate from the CA >>> into the new keystore. >>> >>> The website works but after logging in there are memory leak >>> warnings and the Tomcat service crashes within just a couple of minutes. >>> >>> We even upgraded to 9.0.76 and the issue persists. >>> >>> Below is an excerpt from the stderr log file. >>> >>> I have been unable to find any recent threads on this, any help is >>> appreciated. Is any other information needed? >> >> I think you have included all necessary information. I'm chopping-out >> the irrrelevant bits: >> >>> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd] >>> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to >>> unregister it when the web application was stopped. To prevent a >>> memory leak, the JDBC Driver has been forcibly unregistered. >> >> This is actually "okay" in that Tomcat has detected a global JDBC >> driver registration performed by the application, and is fixing the >> problem for you. The application, however, is making a mistake by not >> de-registering that JDBC driver. Your options are to move the driver >> library from your application into Tomcat's lib/ directory, fix the >> application to de-register the driver when it shuts down, or just ignore the >> warning. >> But you should fix the application. >> >>> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] >>> appears to have started a thread named >>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] >>> but has failed to stop it. This is very likely to create a memory leak. >>> Stack trace of thread: >> >> There are multiple instances of this same message and THIS is your >> problem. The problem is what the error message says: your application >> has started a Thread and never stopped it. The "memory leak" comes >> from the fact that the Thread has inherited the web application's >> ClassLoader and the web application is being re-loaded. When that >> happens, Tomcat discards the ClassLoader which usually means the GC >> will clean up after it at some point in the future. But that Thread >> is still running and will keep the ClassLoader in memory, likely forever. >> >> You have a few options: >> >> 1. Fix the application. The application needs to shut-down any >> threads is starts during its operation, preferrably in a >> ServletContextListener's destroy method or similar. >> >> 2. Don't hot-reload your application. Instead, shut-down the JVM and >> re-start it. Ovbviously, this may have availability implications, but >> then again so does running out of memory and having to bounce the >> JVM, anyway. >> > > I was checking the stacks and saw this: They might not be doing > "hot-deploy" of the app. AFAIK those messages come once someone stops > Tomcat. > >> 2023-07-10T20:52:06.292Z INFOStarting ProtocolHandler >> ["https-openssl-nio-10.2.251.132-443"] >> 2023-07-10T20:52:06.346Z INFOServer startup in [28980] milliseconds >> 2023-07-10T21:35:40.487Z INFOPausing ProtocolHandler >> ["https-openssl-nio-10.2.251.132-443"] >> 2023-07-10T21:35:40.495Z INFOStopping service [Catalina] >> 2023-07-10T21:35:40.910Z INFODestroyed pool : >> |default|lo|-2023-07-10T20-51-53.099285500Z Good point: Tomcat will complain about this stuff on shutdown even if the warnings are irrelevant because the application-stop leak-c
Re: Tomcat 9.0.76 Memory leak with Java 17
Michael, On 7/12/23 07:33, Michael Osipov wrote: On 2023/07/11 18:16:24 Christopher Schultz wrote: You should report all of the previous issues to Oracle against their ORDS version 22.1 and ask them to fix them. It's why you write those big, fat checks in the first place ;) This doesn't really matter. I have reported a memory leak in OJDBC many years ago where a background thread pins the WebappClassLoader. Answer from Oracle: This happens only once, does not repeat. We won't address that... Fair enough, but that doesn't mean it's not worth reporting. BTW the fine folks at Sun/Oracle and Connector/J are similarly confused when it comes to ClassLoader pinning in /their/ JDBC driver. I think they had a few different patches that all waved their hands _still_ failing to actually understand and fix the problem. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.76 Memory leak with Java 17
Suvendu, On 7/12/23 07:11, Suvendu Sekhar Mondal wrote: On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz wrote: James, On 7/11/23 10:21, James Boggs wrote: We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL. We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL csr file, and imported a new SSL certificate from the CA into the new keystore. The website works but after logging in there are memory leak warnings and the Tomcat service crashes within just a couple of minutes. We even upgraded to 9.0.76 and the issue persists. Below is an excerpt from the stderr log file. I have been unable to find any recent threads on this, any help is appreciated. Is any other information needed? I think you have included all necessary information. I'm chopping-out the irrrelevant bits: 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. This is actually "okay" in that Tomcat has detected a global JDBC driver registration performed by the application, and is fixing the problem for you. The application, however, is making a mistake by not de-registering that JDBC driver. Your options are to move the driver library from your application into Tomcat's lib/ directory, fix the application to de-register the driver when it shuts down, or just ignore the warning. But you should fix the application. 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: There are multiple instances of this same message and THIS is your problem. The problem is what the error message says: your application has started a Thread and never stopped it. The "memory leak" comes from the fact that the Thread has inherited the web application's ClassLoader and the web application is being re-loaded. When that happens, Tomcat discards the ClassLoader which usually means the GC will clean up after it at some point in the future. But that Thread is still running and will keep the ClassLoader in memory, likely forever. You have a few options: 1. Fix the application. The application needs to shut-down any threads is starts during its operation, preferrably in a ServletContextListener's destroy method or similar. 2. Don't hot-reload your application. Instead, shut-down the JVM and re-start it. Ovbviously, this may have availability implications, but then again so does running out of memory and having to bounce the JVM, anyway. I was checking the stacks and saw this: They might not be doing "hot-deploy" of the app. AFAIK those messages come once someone stops Tomcat. 2023-07-10T20:52:06.292Z INFOStarting ProtocolHandler ["https-openssl-nio-10.2.251.132-443"] 2023-07-10T20:52:06.346Z INFOServer startup in [28980] milliseconds 2023-07-10T21:35:40.487Z INFOPausing ProtocolHandler ["https-openssl-nio-10.2.251.132-443"] 2023-07-10T21:35:40.495Z INFOStopping service [Catalina] 2023-07-10T21:35:40.910Z INFODestroyed pool : |default|lo|-2023-07-10T20-51-53.099285500Z Good point: Tomcat will complain about this stuff on shutdown even if the warnings are irrelevant because the application-stop leak-checks are run whether Tomcat is going to stop or not. So operationally, these warnings may be irrelevant. (IMHO you should still fix the application!) Another thing I noticed was the difference in allocated Java heap. It appears that first it sets min=max=1GB and after that it's setting max heap to 256MB. That could be the problem. JVM might be crashing because of heap shortage. 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dconfig.url=D:\ords222 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms1024M 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx1024M 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms128m 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx256m Oh, I hadn't noticed that, either. Dropping your heap
Re: Tomcat 9.0.76 Memory leak with Java 17
On 2023/07/11 18:16:24 Christopher Schultz wrote: > You should report all of the previous issues to Oracle against their > ORDS version 22.1 and ask them to fix them. It's why you write those > big, fat checks in the first place ;) This doesn't really matter. I have reported a memory leak in OJDBC many years ago where a background thread pins the WebappClassLoader. Answer from Oracle: This happens only once, does not repeat. We won't address that... - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.76 Memory leak with Java 17
Hi Chris, On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz wrote: > > James, > > On 7/11/23 10:21, James Boggs wrote: > > We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows > > Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL. > > > > We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to > > ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL > > csr file, and imported a new SSL certificate from the CA into the new > > keystore. > > > > The website works but after logging in there are memory leak warnings > > and the Tomcat service crashes within just a couple of minutes. > > > > We even upgraded to 9.0.76 and the issue persists. > > > > Below is an excerpt from the stderr log file. > > > > I have been unable to find any recent threads on this, any help is > > appreciated. Is any other information needed? > > I think you have included all necessary information. I'm chopping-out > the irrrelevant bits: > > > 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd] > > registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to > > unregister it when the web application was stopped. To prevent a memory > > leak, the JDBC Driver has been forcibly unregistered. > > This is actually "okay" in that Tomcat has detected a global JDBC driver > registration performed by the application, and is fixing the problem for > you. The application, however, is making a mistake by not de-registering > that JDBC driver. Your options are to move the driver library from your > application into Tomcat's lib/ directory, fix the application to > de-register the driver when it shuts down, or just ignore the warning. > But you should fix the application. > > > 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] > > appears to have started a thread named > > [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] > > but has failed to stop it. This is very likely to create a memory leak. > > Stack trace of thread: > > There are multiple instances of this same message and THIS is your > problem. The problem is what the error message says: your application > has started a Thread and never stopped it. The "memory leak" comes from > the fact that the Thread has inherited the web application's ClassLoader > and the web application is being re-loaded. When that happens, Tomcat > discards the ClassLoader which usually means the GC will clean up after > it at some point in the future. But that Thread is still running and > will keep the ClassLoader in memory, likely forever. > > You have a few options: > > 1. Fix the application. The application needs to shut-down any threads > is starts during its operation, preferrably in a > ServletContextListener's destroy method or similar. > > 2. Don't hot-reload your application. Instead, shut-down the JVM and > re-start it. Ovbviously, this may have availability implications, but > then again so does running out of memory and having to bounce the JVM, > anyway. > I was checking the stacks and saw this: They might not be doing "hot-deploy" of the app. AFAIK those messages come once someone stops Tomcat. > 2023-07-10T20:52:06.292Z INFOStarting ProtocolHandler > ["https-openssl-nio-10.2.251.132-443"] > 2023-07-10T20:52:06.346Z INFOServer startup in [28980] milliseconds > 2023-07-10T21:35:40.487Z INFOPausing ProtocolHandler > ["https-openssl-nio-10.2.251.132-443"] > 2023-07-10T21:35:40.495Z INFOStopping service [Catalina] > 2023-07-10T21:35:40.910Z INFODestroyed pool : > |default|lo|-2023-07-10T20-51-53.099285500Z Another thing I noticed was the difference in allocated Java heap. It appears that first it sets min=max=1GB and after that it's setting max heap to 256MB. That could be the problem. JVM might be crashing because of heap shortage. > 10-Jul-2023 16:51:35.481 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > -Dconfig.url=D:\ords222 > 10-Jul-2023 16:51:35.481 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > -Xms1024M > 10-Jul-2023 16:51:35.481 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > -Xmx1024M > 10-Jul-2023 16:51:35.486 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > exit > 10-Jul-2023 16:51:35.486 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > abort > 10-Jul-2023 16:51:35.486 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > -Xms128m > 10-Jul-2023 16:51:35.486 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > -Xmx256m > > 2023-07-10T21:35:40.944Z SEVERE The web application [rplans-vpd] > > created a ThreadLocal with key of type > > [java.lang.ThreadLocal.SuppliedThreadLocal] (value > >
Re: Tomcat 9.0.76 Memory leak with Java 17
James, On 7/11/23 10:21, James Boggs wrote: We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL. We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL csr file, and imported a new SSL certificate from the CA into the new keystore. The website works but after logging in there are memory leak warnings and the Tomcat service crashes within just a couple of minutes. We even upgraded to 9.0.76 and the issue persists. Below is an excerpt from the stderr log file. I have been unable to find any recent threads on this, any help is appreciated. Is any other information needed? I think you have included all necessary information. I'm chopping-out the irrrelevant bits: 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. This is actually "okay" in that Tomcat has detected a global JDBC driver registration performed by the application, and is fixing the problem for you. The application, however, is making a mistake by not de-registering that JDBC driver. Your options are to move the driver library from your application into Tomcat's lib/ directory, fix the application to de-register the driver when it shuts down, or just ignore the warning. But you should fix the application. 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: There are multiple instances of this same message and THIS is your problem. The problem is what the error message says: your application has started a Thread and never stopped it. The "memory leak" comes from the fact that the Thread has inherited the web application's ClassLoader and the web application is being re-loaded. When that happens, Tomcat discards the ClassLoader which usually means the GC will clean up after it at some point in the future. But that Thread is still running and will keep the ClassLoader in memory, likely forever. You have a few options: 1. Fix the application. The application needs to shut-down any threads is starts during its operation, preferrably in a ServletContextListener's destroy method or similar. 2. Don't hot-reload your application. Instead, shut-down the JVM and re-start it. Ovbviously, this may have availability implications, but then again so does running out of memory and having to bounce the JVM, anyway. 2023-07-10T21:35:40.944Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. This is actually "okay" in that Tomcat has detected your application's ThreadLocal variables (objects bound to one or more Threads which are owned by the application) which are not being cleaned-up by the application, and it's fixing the problem for you by, over time, killing each of those Threads and replacing them in the Thread pool for you. Your options are to fix the application or to ignore the warning. But you should fix the application. It appears that your upgrade of ORDS has introduced a lot of stuff that doesn't play well with hot-reloading of the application. I'm assuming that you aren't responsible for maintaining ORDS... Oracle is. You should report all of the previous issues to Oracle against their ORDS version 22.1 and ask them to fix them. It's why you write those big, fat checks in the first place ;) -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 9.0.76 Memory leak with Java 17
Hi, We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL. We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL csr file, and imported a new SSL certificate from the CA into the new keystore. The website works but after logging in there are memory leak warnings and the Tomcat service crashes within just a couple of minutes. We even upgraded to 9.0.76 and the issue persists. Below is an excerpt from the stderr log file. I have been unable to find any recent threads on this, any help is appreciated. Is any other information needed? -- start of logfile 2023-07-10 16:51:33 Apache Commons Daemon procrun stderr initialized. 10-Jul-2023 16:51:35.434 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.76 10-Jul-2023 16:51:35.449 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Jun 5 2023 07:17:04 UTC 10-Jul-2023 16:51:35.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.76.0 10-Jul-2023 16:51:35.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Windows Server 2012 R2 10-Jul-2023 16:51:35.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:6.3 10-Jul-2023 16:51:35.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64 10-Jul-2023 16:51:35.452 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: D:\JAVA\JDK 10-Jul-2023 16:51:35.452 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 17.0.7+8-LTS-224 10-Jul-2023 16:51:35.452 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation 10-Jul-2023 16:51:35.452 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: D:\tomcat9 10-Jul-2023 16:51:35.454 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: D:\tomcat9 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dconfig.url=D:\ords222 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms1024M 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx1024M 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=D:\tomcat9 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=D:\tomcat9 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=D:\tomcat9\endorsed 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=D:\tomcat9\temp 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=D:\tomcat9\conf\logging.properties 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms128m 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx256m 10-Jul-2023 16:51:35.496 INFO [main]
