Hi Mark, thank you for providing those steps. I downloaded a "YourKit" trial and did execute them.
I am blocked at the "Figure out what is creating those reference chains" step. The documentation [ https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/catalina/loader/WebappClassLoaderBase.html ] states the following implementation note: """ By default, this class loader follows the delegation model required by the specification. The bootstrap class loader will be queried first, then the local repositories, and only then delegation to the parent class loader will occur. This allows the web application to override any shared class except the classes from J2SE. Special handling is provided from the JAXP XML parser interfaces, the JNDI interfaces, and the classes from the servlet API, which are never loaded from the webapp repositories. The delegate property allows an application to modify this behavior to move the parent class loader ahead of the local repositories. """ Not sure I am interpreting the doc correctly, does this mean that the concerned classes of the xercesImpl jar in <context>/WEB-INF/lib will be ignored when there exists the equivalent in the bootstrap class loader and/or the parent one if any ? So the problem I am experiencing is with JAXP XML parser interfaces, specifically I am getting an error when our code calls javax.xml.bind.DatatypeConverter.parseDateTime(someString) so that the org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl tries to use org.apache.xerces.jaxp.datatype.XMLGregorianCalendarImpl but cannot load the inner class called XMLGregorianCalendarImpl$DaysInMonth. Opening the HPROF memory dump, I've found that after context is stopped, the org.apache.xerces.jaxp.datatype.XMLGregorianCalendarImpl class is referenced in 2 ParallelWebappClassLoader although almost all contexts that are subject to be using some xml were stopped except Tomcat's manager. 1) GC Root from one of our class that has a static method that calls SSLContext.setDefault() with an ssl context declared inside that static method as parameter. 2) GC Root from a sun.awt.AppContext. We are not using AppContext directly so I blocked there. Does that finding of 2 ParallelWebappClassLoader was the right thing to find ? I would have tought that the reference I was looking for would have been in either the <SystemClassLoader> or the java.net.URLClassLoader that contains most of the used catalina classes. Moreover, I've found that there is one of the existing org.apache.jasper.servlet.JasperLoader that contains the compiled JSP class that is faulty, could that be related somehow? I've changed the SSLContext.setDefault call from one of our class so it's not dangling anymore and run the steps one more time, stopping all contexts this time except tomcat's manager and opening the HPROF dump found: 1) XMLGregorianCalendarImpl in classes of org.apache.catalina.loader.ParallelWebappClassLoader [Held by JVM] 2) XMLGregorianCalendarImpl from constant pool of org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl which is the datatypeFactory of com.sun.xml.bind.DatatypeConverterImpl which is the theConverter of javax.xml.bind.DatatypeConverter which itself is a GC Root. Could I for instance add a Class.forName(javax.xml.bind.DatatypeConverter) in my JSP or perhaps try to explicitly load that class differently? At this point, I do not know what to do next! Simon On Thu, Apr 24, 2025 at 5:40 AM Mark Thomas <ma...@apache.org> wrote: > On 23/04/2025 16:10, Simon Arame wrote: > > <snip/> > > > What java method call should I be searching for in libraries source code > ? > > It isn't going to be that simple. You'll need to do something like. > > 1. Deploy the application > > 2. Undeploy the application > > 3. Force GC (with a profiler) > > 4. Take a memory snapshot (with a profiler) > > 5. Look for instances of the web application class loader > > 6. Find the one with the state STOPPED or similar > > 7. Trace the GC roots > > 8. Figure out what is creating those reference chains > > 9. Figure out what you need to do to destroy each reference chain when > the application shuts down and/or cause each reference chain to be > created to the common class loader rather than the web application > specific one. > > If you can provide a minimal web application that reproduces the issue, > we can help with the above. > > Mark > > > > > > > Simon > > > > On Tue, Apr 22, 2025 at 12:02 PM Mark Thomas <ma...@apache.org> wrote: > > > >> On 22/04/2025 16:44, Simon Arame wrote: > >> > >> <snip/> > >> > >>> What is strange is that although it says "this web application instance > >> has > >>> been stopped already", the web application is still running, end users > >> are > >>> still receiving 200 OKs from the web application. > >> > >> Any other web applications running on that Tomcat instance? > >> > >> Has the web application with the issue ever been restarted / reloaded? > >> > >>> We are not sure what causes this because it does not always happen, it > is > >>> "intermittent". The SoapBindingStub is called from a JSP which is aimed > >> to > >>> simulate a call to a Soap service as if it was coming from outside of > >> this > >>> app. > >>> So the jsp that made the call received this 500 error from the Soap > >> service > >>> but this is unusual. > >> > >> What you are seeing is the web application attempting to use classes > >> that were loaded by a different/"older version of the current" web > >> application. > >> > >> This sort of thing can happen when a web application (or a library is > >> uses) registers something at the JVM level which then becomes a global > >> resource rather than a web application specific resource. > >> > >> The trick to fixing this is figuring out what code is performing this > >> JVM level registration and moving that code from the web application to > >> $CATALINA_BASE/lib > >> > >> If it is a library the application is using, this is relatively simple. > >> If it is the application then things get trickier. > >> > >> Mark > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
