On 7/11/23 10:21, James Boggs wrote:
We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.

We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL csr file, and imported a new SSL certificate from the CA into the new keystore.

The website works but after logging in there are memory leak warnings and the Tomcat service crashes within just a couple of minutes.

We even upgraded to 9.0.76 and the issue persists.

Below is an excerpt from the stderr log file.

I have been unable to find any recent threads on this, any help is appreciated. Is any other information needed?

I think you have included all necessary information. I'm chopping-out the irrrelevant bits:

2023-07-10T21:35:40.939Z WARNING     The web application [rplans-vpd] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

This is actually "okay" in that Tomcat has detected a global JDBC driver registration performed by the application, and is fixing the problem for you. The application, however, is making a mistake by not de-registering that JDBC driver. Your options are to move the driver library from your application into Tomcat's lib/ directory, fix the application to de-register the driver when it shuts down, or just ignore the warning. But you should fix the application.

2023-07-10T21:35:40.944Z WARNING     The web application [rplans-vpd] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

There are multiple instances of this same message and THIS is your problem. The problem is what the error message says: your application has started a Thread and never stopped it. The "memory leak" comes from the fact that the Thread has inherited the web application's ClassLoader and the web application is being re-loaded. When that happens, Tomcat discards the ClassLoader which usually means the GC will clean up after it at some point in the future. But that Thread is still running and will keep the ClassLoader in memory, likely forever.

You have a few options:

1. Fix the application. The application needs to shut-down any threads is starts during its operation, preferrably in a ServletContextListener's destroy method or similar.

2. Don't hot-reload your application. Instead, shut-down the JVM and re-start it. Ovbviously, this may have availability implications, but then again so does running out of memory and having to bounce the JVM, anyway.

2023-07-10T21:35:40.944Z SEVERE      The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

This is actually "okay" in that Tomcat has detected your application's ThreadLocal variables (objects bound to one or more Threads which are owned by the application) which are not being cleaned-up by the application, and it's fixing the problem for you by, over time, killing each of those Threads and replacing them in the Thread pool for you. Your options are to fix the application or to ignore the warning. But you should fix the application.

It appears that your upgrade of ORDS has introduced a lot of stuff that doesn't play well with hot-reloading of the application. I'm assuming that you aren't responsible for maintaining ORDS... Oracle is.

You should report all of the previous issues to Oracle against their ORDS version 22.1 and ask them to fix them. It's why you write those big, fat checks in the first place ;)


To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Reply via email to