Hello William, On Tue, Nov 7, 2023 at 4:29 PM William Crowell <wcrow...@perforce.com.invalid> wrote: > > Olaf and Sevendu, > > Thank you for your replies. Correct, I sincerely doubt this is a Tomcat > class loading bug. > > I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into > memory, and it is a single class loader. > > I am going to periodically run: jcmd <pid> GC.class_stats > > I am only having the issue in general operation and not on a redeploy, and I > have to restart the server daily. I will find out more details as to how > these classes are loaded into memory. > > Here is what is going on… > > I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on > it. It is running JDK 1.8.0_371-b25. I have min and max JVM heap setting at > 8GB. JVM arguments are: > > -XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC > -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 > -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 > -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 > -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 > -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC > -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime > -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > -XX:+UseCompressedClassPointers -XX:+UseCompressedOops > -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation > > The MaxMetaspaceSize is not set, so this means it is unlimited. > > After the server comes up and after a short period of time I get a native out > of memory condition: > > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (mmap) failed to map 8589934592 bytes for > committing reserved memory. > # Possible reasons: > # The system is out of physical RAM or swap space > # The process is running with CompressedOops enabled, and the Java Heap may > be blocking the growth of the native heap > # Possible solutions: > # Reduce memory load on the system > # Increase physical memory or swap space > # Check if swap backing store is full > # Decrease Java heap size (-Xmx/-Xms) > # Decrease number of Java threads > # Decrease Java thread stack sizes (-Xss) > # Set larger code cache with -XX:ReservedCodeCacheSize= > # This output file may be truncated or incomplete. > # > # Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x000014fff94b7700 > # > # JRE version: (8.0_371) (build ) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode > linux-amd64 compressed oops) > # Core dump written. Default location: /hosting/core or core.191803 > # >
So, it is not a Metaspace OOM. It can happen that Metaspace or some other segment of JVM is increasing with time. Since you have NMT enabled(-XX:NativeMemoryTracking=detail), please collect native memory stats periodically until JVM crashes. After that, if you compare them, you should see one of the segments is growing with time. Then you can focus on that area. jcmd <pid> VM.native_memory can help with this. > Regards, > > William Crowell > > > From: Olaf Kock <tom...@olafkock.de> > Date: Tuesday, November 7, 2023 at 4:25 AM > To: users@tomcat.apache.org <users@tomcat.apache.org> > Subject: Re: Are there any known class loader leaks in Tomcat 9? > > On 06.11.23 18:55, William Crowell wrote: > > Good afternoon, > > > > I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am > > loading some very large Java classes into Metaspace. I know this is not > > good practice, but I inherited this library. These classes have business > > rules and are doing some basic primitive and array manipulations, and I am > > running out of native memory. I don’t think there are any JNI calls in > > this code base. > > > > Are there anything existing issues with the Tomcat 9 class loader? I doubt > > there are but wanted to check. > > Hi William, > > when you say "I am loading some very large Java classes into Metaspace. > I know this is not good practice" it does not sound like this inherited > library is using tomcat's standard way of classloading - please clarify. > > Also, nothing in that statement points to anything where a leak on > tomcat's side would make a difference: Are you experiencing problems > after reloading a webapp, or just in general operation? You might simply > need to assign more metaspace to the VM for this library to operate > properly. > > So, this boils down to: > > * How does this library load the classes into memory? > > * Do you redeploy? > > * What's your metaspace (and general memory) setting, and what are the > conditions under which you run out of memory? > > * Do you see any log message that hint at classes that won't be garbage > collected because of stale references? Tomcat issues those warnings, in > case you (for example) start your own background threads that keep > holding the memory allocated to them. > > Also: I second your doubt about Tomcat's classloader being the cause for > your problems - unless proven otherwise. > > Olaf > > > > > CAUTION: This email originated from outside of the organization. Do not click > on links or open attachments unless you recognize the sender and know the > content is safe. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
