Hi Karol, The errors you are sharing still aren't making sense to me. They all seem to reference Tomcat having issues... and none of them reference DSpace. Are you certain you are using a supported version of Tomcat? https://wiki.lyrasis.org/display/DSDOC7x/Installing+DSpace#InstallingDSpace-ServletEngine(ApacheTomcat9,Jetty,CauchoResinorequivalent)
Are you running anything else in Tomcat besides DSpace? Have you tried restarting Tomcat to see if the behavior continues (or see if errors are shown in the Tomcat logs during the restart)? What are your CATALINA_OPTS settings that you configured via https://wiki.lyrasis.org/display/DSDOC7x/Performance+Tuning+DSpace ? Basically, based on what you've shared, it all seems to imply there's something wrong with your Tomcat installation. I'm not sure what could be wrong, but you may want to look more closely there. You also could try installing Tomcat over again to see if something went wrong with the Tomcat install (even small things like permissions issues can sometimes cause problems in Tomcat). I wish I had better ideas, but I've never seen this issue before. Tim On Friday, September 1, 2023 at 4:46:18 AM UTC-5 [email protected] wrote: > Hi Tim, > > yes, I used > https://wiki.lyrasis.org/display/DSDOC7x/Performance+Tuning+DSpace for > increase performance. I found something like this: > > [2023-09-01 10:03:55] [warning] Failed to clear soft references from > ObjectStreamClass$Caches for web application [ROOT] > [2023-09-01 10:03:55] [warning] java.lang.ClassCastException: class > java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map > (java.io.ObjectStreamClass$Caches$1 and java.util.Map are in module > java.base of loader 'bootstrap') > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.loader.WebappClassLoaderBase.clearCache(WebappClassLoaderBase.java:2282) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches(WebappClassLoaderBase.java:2257) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.loader.WebappClassLoaderBase.clearReferences(WebappClassLoaderBase.java:1626) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.loader.WebappClassLoaderBase.stop(WebappClassLoaderBase.java:1554) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:461) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5469) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389) > [2023-09-01 10:03:55] [warning] at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > [2023-09-01 10:03:55] [warning] at > org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) > [2023-09-01 10:03:55] [warning] at > java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389) > [2023-09-01 10:03:55] [warning] at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > [2023-09-01 10:03:55] [warning] at > org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) > [2023-09-01 10:03:55] [warning] at > java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.core.StandardService.stopInternal(StandardService.java:498) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:982) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.startup.Catalina.stop(Catalina.java:849) > [2023-09-01 10:03:55] [warning] at > org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:996) > [2023-09-01 10:03:56] [warning] java.lang.ClassCastException: class > java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map > (java.io.ObjectStreamClass$Caches$1 and java.util.Map are in module > java.base of loader 'bootstrap') > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.loader.WebappClassLoaderBase.clearCache(WebappClassLoaderBase.java:2282) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches(WebappClassLoaderBase.java:2257) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.loader.WebappClassLoaderBase.clearReferences(WebappClassLoaderBase.java:1626) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.loader.WebappClassLoaderBase.stop(WebappClassLoaderBase.java:1554) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:461) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5469) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389) > [2023-09-01 10:03:56] [warning] at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > [2023-09-01 10:03:56] [warning] at > org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) > [2023-09-01 10:03:56] [warning] at > java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389) > [2023-09-01 10:03:56] [warning] at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > [2023-09-01 10:03:56] [warning] at > org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) > [2023-09-01 10:03:56] [warning] at > java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.core.StandardService.stopInternal(StandardService.java:498) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:982) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.startup.Catalina.stop(Catalina.java:849) > [2023-09-01 10:03:56] [warning] at > org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:996) > [2023-09-01 10:03:56] [warning] The web application [server] appears to > have started a thread named [Thread-4] but has failed to stop it. This is > very likely to create a memory leak. Stack trace of thread: > [2023-09-01 10:03:56] [info] [email protected]/java.lang.Object.wait(Native > Method) > [2023-09-01 10:03:56] [info] > > org.dspace.service.impl.HttpConnectionPoolService$IdleConnectionMonitorThread.run(HttpConnectionPoolService.java:176) > [2023-09-01 10:03:56] [warning] Failed to clear soft references from > ObjectStreamClass$Caches for web application [server] > [2023-09-01 10:03:56] [warning] java.lang.ClassCastException: class > java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map > (java.io.ObjectStreamClass$Caches$1 and java.util.Map are in module > java.base of loader 'bootstrap') > > Best, > > Karol > czwartek, 31 sierpnia 2023 o 19:09:49 UTC+2 DSpace Technical Support > napisał(a): > >> Hi Karol, >> >> Based on the details you've shared, it's really difficult to tell what >> may be happening. It sounds like Tomcat is using your swap, so that'd >> imply that it is overwhelmed. This could be that the DSpace backend is >> overwhelmed at that time, or maybe something in Tomcat isn't "tuned" or >> configured properly? >> >> Have you followed the performance tuning guide (especially the parts for >> Tomcat)? >> https://wiki.lyrasis.org/display/DSDOC7x/Performance+Tuning+DSpace >> >> Are there any other clues in your logs or the Tomcat logs which might >> provide more information (the logs you shared aren't very >> informative...they just show that swap is likely occurring, but not clear >> why)? Any errors you see in those logs? Here's our troubleshooting guide >> for help on locating errors: >> >> https://wiki.lyrasis.org/display/DSPACE/Troubleshoot+an+error#Troubleshootanerror-DSpace7.x(orabove) >> >> Basically, it's unclear to me what could be going on. So, it'd be very >> helpful if you could search for more clues & double check that you've >> "tuned" Tomcat via CATALINA_OPTS or similar. >> >> Tim >> >> On Monday, August 28, 2023 at 2:21:27 AM UTC-5 [email protected] wrote: >> >>> Hi Bram, >>> >>> it's happening on DSpace 7.5, angular 7.5. My environment: >>> >>> - Debian 11 >>> - Java version: 11.0.20 (java-11-openjdk) >>> - Apache Tomcat/9.0.43 >>> - pm2 version 5.3.0 started with cluster mode >>> >>> Thanks, >>> >>> Karol >>> >>> >>> >>> niedziela, 27 sierpnia 2023 o 16:16:50 UTC+2 Bram Luyten napisał(a): >>> >>>> Hi Karol, >>>> >>>> On which version of DSpace do you see this happening? >>>> >>>> Cheers, >>>> >>>> Bram >>>> >>>> [image: logo] Bram Luyten >>>> *1335 Jefferson Road # 23282, Rochester, NY 14692* >>>> Gaston Geenslaan 14, 3001 Leuven, Belgium >>>> DSpace Express Hosting >>>> <https://www.atmire.com/dspace-express?utm_source=emailfooter&utm_medium=email&utm_campaign=braml> >>>> - Open Repository Hosting >>>> <https://www.atmire.com/open-repository?utm_source=emailfooter&utm_medium=email&utm_campaign=braml> >>>> - Custom DSpace Services >>>> <https://www.atmire.com/custom-dspace?utm_source=emailfooter&utm_medium=email&utm_campaign=braml> >>>> >>>> >>>> On Sat, 26 Aug 2023 at 15:03, Karol <[email protected]> wrote: >>>> >>>>> hi, >>>>> >>>>> I have a lot of memory allocated for dspace 32,64,128 Gb - it doesn't >>>>> matter, swap is in use anyway. I found such errors in tomcat: >>>>> >>>>> The web application [server] created a ThreadLocal with key of type >>>>> [java.lang.ThreadLocal.SuppliedThreadLocal] (value >>>>> [java.lang.ThreadLocal$SuppliedThreadLocal@4e12dfd8]) and a value of type >>>>> [org.springframework.util.ConcurrentLruCache] (value >>>>> [org.springframework.util.ConcurrentLruCache@4cf0f8fd]) but failed to >>>>> remove it when the web application was stopped. Threads are going to be >>>>> renewed over time to try and avoid a probable memory leak >>>>> >>>>> And swap is used by the process: >>>>> /usr/lib/jvm/default-java/bin/java >>>>> -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties >>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >>>>> >>>>> Greetings, >>>>> >>>>> Karol >>>>> >>>>> -- >>>>> All messages to this mailing list should adhere to the Code of >>>>> Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "DSpace Technical Support" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/dspace-tech/ac1e66bb-21ef-40e9-88b1-421f1e313211n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/dspace-tech/ac1e66bb-21ef-40e9-88b1-421f1e313211n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/0152ba18-408c-48a6-aa0e-2f23941c2ce7n%40googlegroups.com.
