Thanks for the reply Thomas. We are getting a OOME on metaspace and do not set CompressedClassSpaceSize. It seems our used CompressedClassSpaceSize is ~100MB. If this isn't set, will it use as much of the specified XX:MaxMetaspaceSize as needed?
We do have a lot of class loaders, after looking closely at the heap. Looks like ~11k loaders, where the majority of them are sun.reflect.DelegatingClassLoader's corresponding to reflective method instances that are being strongly referenced. I'm not sure if the chunk size is 64k because that would lead to ~687MB of Metaspace going to the initial allocation for each loader, however looking at the output of "jcmd <pid> GC.class_stats" and summing up the total for all DelegatingClassLoader's it shows only ~46MB is used and the remaining classes account for ~840MB. Maybe the total accumulated chunk memory is not part of the output of "jcmd <pid> GC.class_stats", do you know if allocated by unused space by the loader is reported here? If not is there anyway to get this info on a production JVM? Looking at the pdf was on the reference of 2, it seem like the DelegatingClassLoader's are consuming 4 small chunks of 512 words which is 4 x 512 words x 8 bytes/word so as a lower bound i would expect the cost of all those loaders to be ~171MB. The metaspace usage in the JVM is ~940MB and compressed class space is ~100mb, so adding the ~171MB it seems to bring us to a total of ~1.18G which is roughly the value of XX:MaxMetaspaceSize. I've found a few ways to limit the number of DelegatingClassLoader's by either changing the inflation threshold or we could possibly just break the path back to gc root of the class instance that is owned by the corresponding DelegatingClassLoader. It's a shame that the same DelegatingClassLoader isn't reused, but I suppose the finer granularity of method to classloader is to increase the chances that the loader can be unloaded when the method is not longer referenced. On Tue, Oct 25, 2016 at 2:37 AM, Thomas Stüfe <thomas.stu...@gmail.com> wrote: > Hi Brian, > > On Sat, Oct 22, 2016 at 1:55 AM, Brian Toal <brian.t...@gmail.com> wrote: > >> Good evening. In a application that I'm responsible for, Metaspace is >> set to 1.1GB. Specifically the following flags are set: >> >> -XX:MetaspaceSize=1152m >> -XX:MaxMetaspaceSize=1152m >> -XX:MinMetaspaceFreeRatio=0 >> -XX:MaxMetaspaceFreeRatio=100 >> >> However we are getting a OOME when metaspace size hits 80% of 1.1GB. >> > > Out of metaspace or out of compressed class space? If the latter, have you > set CompressedClassSpaceSize? > > >> Doing a bit or research it seems that Metaspace is known to fragement the >> memory when a loader needs to acquire memory from the current chunk, and >> the current chuck can't accomodate the request, the pointer is bumped to >> the next available chunk, meaning any free memory in the previous chunks >> block is gone with the wind. >> > > No. The remaining space is put into freelists (both on chunk and on block > level) and used for follow-up requests, should the size fit. In our > experience, we see very little wastage due to "half-eaten blocks/chunks. > > There are other possible waste scenarios: > > 1) you have a lot of class loaders living in parallel. Each one will take > a chunk of memory (its current chunk) and satisfy memory requests from > there. This means that the current chunk always contains a portion of > still-unused memory which may be used by this class loader in the future > but already counts against MaxMetaspaceSize. However, to make this hurt, > you really have to have very many class loaders in parallel, as the maximum > possible overhead for this scenario cannot exceed the size of a medium > chunk per classloader (64k?) > > 2) The scenario described in this JEP here: https://bugs.openjdk. > java.net/browse/JDK-8166690 . > > 3) real fragmentation (i.e. a mixture of in-use and free chunks). > > In my practice, I keep seeing (2). Hence the JEP, which will hopefully > help. > > Kind Regards, Thomas > > > >> More than happy if someone corrects my understand here or can point me to >> a good reference that explains this in detail. >> >> My question is, how to do I monitor current usage + fragmentation so the >> proportion of free space can be monitored? >> >> Also is there any tuning that can take place to reduce the proportion of >> fragmentation? >> >> Does compressed cache acquire memory from memory set aside from memory >> allocated via MaxMetaspaceSize? >> >> Thanks in advance. >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use@openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >
_______________________________________________ hotspot-gc-use mailing list hotspot-gc-use@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
