Re: Are there any known class loader leaks in Tomcat 9?
William, On 11/7/23 05:59, William Crowell wrote: Olaf and Sevendu, Thank you for your replies. Correct, I sincerely doubt this is a Tomcat class loading bug. I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into memory, and it is a single class loader. I am going to periodically run: jcmd GC.class_stats I am only having the issue in general operation and not on a redeploy, and I have to restart the server daily. I will find out more details as to how these classes are loaded into memory. Here is what is going on… I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on it. It is running JDK 1.8.0_371-b25. I have min and max JVM heap setting at 8GB. JVM arguments are: -XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation The MaxMetaspaceSize is not set, so this means it is unlimited. After the server comes up and after a short period of time I get a native out of memory condition: # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x14fff94b7700 # # JRE version: (8.0_371) (build ) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode linux-amd64 compressed oops) # Core dump written. Default location: /hosting/core or core.191803 # Enable heap dumps on OOME and look at them with a heap analyzer. You may find that you have some huge things in memory that aren't being released for other reasons. If you have to bounce your server every day, I suspect that you have a "known resource leak" in your web application already. It's very unlikely to be Tomcat doing this. What you should NOT do is just keep raising the heap size until it stops failing, because you will never find out what is taking up all that heap space. If you find that nothing is out of place in that heap dump then AND ONLY then should you raise the heap size. Sometimes you really just need more heap. -chri - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Are there any known class loader leaks in Tomcat 9?
Hello William, On Tue, Nov 7, 2023 at 4:29 PM William Crowell wrote: > > Olaf and Sevendu, > > Thank you for your replies. Correct, I sincerely doubt this is a Tomcat > class loading bug. > > I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into > memory, and it is a single class loader. > > I am going to periodically run: jcmd GC.class_stats > > I am only having the issue in general operation and not on a redeploy, and I > have to restart the server daily. I will find out more details as to how > these classes are loaded into memory. > > Here is what is going on… > > I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on > it. It is running JDK 1.8.0_371-b25. I have min and max JVM heap setting at > 8GB. JVM arguments are: > > -XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC > -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 > -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 > -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 > -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 > -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC > -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime > -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > -XX:+UseCompressedClassPointers -XX:+UseCompressedOops > -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation > > The MaxMetaspaceSize is not set, so this means it is unlimited. > > After the server comes up and after a short period of time I get a native out > of memory condition: > > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (mmap) failed to map 8589934592 bytes for > committing reserved memory. > # Possible reasons: > # The system is out of physical RAM or swap space > # The process is running with CompressedOops enabled, and the Java Heap may > be blocking the growth of the native heap > # Possible solutions: > # Reduce memory load on the system > # Increase physical memory or swap space > # Check if swap backing store is full > # Decrease Java heap size (-Xmx/-Xms) > # Decrease number of Java threads > # Decrease Java thread stack sizes (-Xss) > # Set larger code cache with -XX:ReservedCodeCacheSize= > # This output file may be truncated or incomplete. > # > # Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x14fff94b7700 > # > # JRE version: (8.0_371) (build ) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode > linux-amd64 compressed oops) > # Core dump written. Default location: /hosting/core or core.191803 > # > So, it is not a Metaspace OOM. It can happen that Metaspace or some other segment of JVM is increasing with time. Since you have NMT enabled(-XX:NativeMemoryTracking=detail), please collect native memory stats periodically until JVM crashes. After that, if you compare them, you should see one of the segments is growing with time. Then you can focus on that area. jcmd VM.native_memory can help with this. > Regards, > > William Crowell > > > From: Olaf Kock > Date: Tuesday, November 7, 2023 at 4:25 AM > To: users@tomcat.apache.org > Subject: Re: Are there any known class loader leaks in Tomcat 9? > > On 06.11.23 18:55, William Crowell wrote: > > Good afternoon, > > > > I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am > > loading some very large Java classes into Metaspace. I know this is not > > good practice, but I inherited this library. These classes have business > > rules and are doing some basic primitive and array manipulations, and I am > > running out of native memory. I don’t think there are any JNI calls in > > this code base. > > > > Are there anything existing issues with the Tomcat 9 class loader? I doubt > > there are but wanted to check. > > Hi William, > > when you say "I am loading some very large Java classes into Metaspace. > I know this is not good practice" it does not sound like this inherited > library is using tomcat's standard way of classloading - please clarify. > > Also, nothing in that statement points to anything where a leak on > tomcat's side would make a difference: Are you experiencing problems > after reloading a webapp, or just in general operation? You might simply > need to assign more metaspace to the VM for this library to operate > properly. > > So, this boils down to: > > * How does this library load the classes into memory? > > * Do you redeploy? > > * What's your metaspace (and general memory) setting, and what are
Re: Are there any known class loader leaks in Tomcat 9?
Olaf and Sevendu, Thank you for your replies. Correct, I sincerely doubt this is a Tomcat class loading bug. I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into memory, and it is a single class loader. I am going to periodically run: jcmd GC.class_stats I am only having the issue in general operation and not on a redeploy, and I have to restart the server daily. I will find out more details as to how these classes are loaded into memory. Here is what is going on… I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on it. It is running JDK 1.8.0_371-b25. I have min and max JVM heap setting at 8GB. JVM arguments are: -XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation The MaxMetaspaceSize is not set, so this means it is unlimited. After the server comes up and after a short period of time I get a native out of memory condition: # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x14fff94b7700 # # JRE version: (8.0_371) (build ) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode linux-amd64 compressed oops) # Core dump written. Default location: /hosting/core or core.191803 # Regards, William Crowell From: Olaf Kock Date: Tuesday, November 7, 2023 at 4:25 AM To: users@tomcat.apache.org Subject: Re: Are there any known class loader leaks in Tomcat 9? On 06.11.23 18:55, William Crowell wrote: > Good afternoon, > > I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am > loading some very large Java classes into Metaspace. I know this is not good > practice, but I inherited this library. These classes have business rules > and are doing some basic primitive and array manipulations, and I am running > out of native memory. I don’t think there are any JNI calls in this code > base. > > Are there anything existing issues with the Tomcat 9 class loader? I doubt > there are but wanted to check. Hi William, when you say "I am loading some very large Java classes into Metaspace. I know this is not good practice" it does not sound like this inherited library is using tomcat's standard way of classloading - please clarify. Also, nothing in that statement points to anything where a leak on tomcat's side would make a difference: Are you experiencing problems after reloading a webapp, or just in general operation? You might simply need to assign more metaspace to the VM for this library to operate properly. So, this boils down to: * How does this library load the classes into memory? * Do you redeploy? * What's your metaspace (and general memory) setting, and what are the conditions under which you run out of memory? * Do you see any log message that hint at classes that won't be garbage collected because of stale references? Tomcat issues those warnings, in case you (for example) start your own background threads that keep holding the memory allocated to them. Also: I second your doubt about Tomcat's classloader being the cause for your problems - unless proven otherwise. Olaf CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Are there any known class loader leaks in Tomcat 9?
On 06.11.23 18:55, William Crowell wrote: Good afternoon, I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am loading some very large Java classes into Metaspace. I know this is not good practice, but I inherited this library. These classes have business rules and are doing some basic primitive and array manipulations, and I am running out of native memory. I don’t think there are any JNI calls in this code base. Are there anything existing issues with the Tomcat 9 class loader? I doubt there are but wanted to check. Hi William, when you say "I am loading some very large Java classes into Metaspace. I know this is not good practice" it does not sound like this inherited library is using tomcat's standard way of classloading - please clarify. Also, nothing in that statement points to anything where a leak on tomcat's side would make a difference: Are you experiencing problems after reloading a webapp, or just in general operation? You might simply need to assign more metaspace to the VM for this library to operate properly. So, this boils down to: * How does this library load the classes into memory? * Do you redeploy? * What's your metaspace (and general memory) setting, and what are the conditions under which you run out of memory? * Do you see any log message that hint at classes that won't be garbage collected because of stale references? Tomcat issues those warnings, in case you (for example) start your own background threads that keep holding the memory allocated to them. Also: I second your doubt about Tomcat's classloader being the cause for your problems - unless proven otherwise. Olaf
Re: Are there any known class loader leaks in Tomcat 9?
Hello William, On Mon, Nov 6, 2023 at 11:25 PM William Crowell wrote: > > Good afternoon, > > I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am > loading some very large Java classes into Metaspace. I know this is not good > practice, but I inherited this library. These classes have business rules > and are doing some basic primitive and array manipulations, and I am running > out of native memory. I don’t think there are any JNI calls in this code > base. > How are you loading those large classes? using some custom classloader OR Tomcat's normal classloader? Also, are you using a single classloader to load all/multiple classes? Reason I asked is, until ALL classes loaded by a classloader are de-referenced, the entire set of classes loaded by the classloader will NOT be garbage collected. Most possibly that is what is happening in your case. Also, how fast Metaspace is growing? I will suggest checking the contents of Metaspace by taking class stats details periodically. GC.class_stats of Jcmd is helpful. > Are there anything existing issues with the Tomcat 9 class loader? I doubt > there are but wanted to check. > I have been using Tomcat 9 for the last two years and have not experienced any problem where Tomcat is holding onto classes. > Regards, > > William Crowell > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Are there any known class loader leaks in Tomcat 9?
Good afternoon, I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am loading some very large Java classes into Metaspace. I know this is not good practice, but I inherited this library. These classes have business rules and are doing some basic primitive and array manipulations, and I am running out of native memory. I don’t think there are any JNI calls in this code base. Are there anything existing issues with the Tomcat 9 class loader? I doubt there are but wanted to check. Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.