Thanks a lot for all the hints! They helped me a lot. I think I'm moving forward. The key thing was to calculate the amount of occupied memory seen by CGroups. It can be easily done using:
- /sys/fs/cgroup/memory/memory.usage_in_bytes - /sys/fs/cgroup/memory/memory.limit_in_bytes Calculated ratio along with Native Memory Tracking [1] helped me to find a good balance. I also found a shortcut which makes setting initial parameters much easier: -XX:MaxRAM [2] (and set it based on CGroups limit). The downside is that with MaxRAM parameter I lose control over Xms. [1] https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html [2] https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/ On Thursday, 3 August 2017 20:16:50 UTC+2, Tom Lee wrote: > > Hey Sebastian, > > Dealt with a similar issues on Docker a few years back -- safest way to do > it is to use some sort of heuristic for your maximum JVM process size. > Working from a very poor memory and perhaps somebody here will tell me this > is a bad idea for perfectly good reasons, but iirc the ham-fisted heuristic > we used at the time for max total JVM process size was something like: > > <runtime value of -Xmx> + <runtime value of -XX:MaxDirectMemorySize> + slop > > Easy enough to see these values via -XX:+PrintFlagsFinal if they're not > explicitly defined by your apps. We typically had Xmx somewhere between > 8-12GB, but MaxDirectMemorySize varied greatly from app to app. Sometimes a > few hundred MB, in some weird cases it was multiples of the JVM heap size. > > The "slop" was for things we hadn't accounted for, but we really should > have included things like the code cache size etc. as Meg's estimate above > does. I think we used ~10% of the JVM heap size, which was probably > slightly wasteful, but worked well enough for us. Suggest you take the > above heuristic and mix it up with Meg's idea to include code cache size > etc. & feel your way from there. I'd personally always leave at least a few > hundred megs additional overhead on top of my "hard" numbers because I > don't trust myself with such things. :) > > Let's see, what else. At the time our JVM -- think this was an Oracle Java > 8 JDK -- set MaxDirectMemorySize to the value of Xmx by default, implying > the JVM process *could* (but not necessarily *would*) grow up to roughly > double its configured size to accommodate heap + direct buffers if you had > an application that made heavy use of direct buffers and put enough > pressure on the heap to grow it to the configured Xmx value (or as we > typically did, set Xmx == Xms). > > Where possible we would constrain MaxDirectMemorySize to something "real" > rather than leaving it to this default, preferring to have the JVM throw up > an OOME if we were allocating more direct memory than we expected so we > could get more info about the failure rather than worrying about the OOM > killer hard kill the entire process & not being able to understand why. > YMMV. > > One caveat: I can't quite remember > if Unsafe.allocateMemory()/Unsafe.freeMemory() count toward your > MaxDirectMemorySize ... perhaps somebody else here more familiar with the > JVM internals could weigh in on that. Perhaps another thing to watch out > for if you're doing "interesting" things with the JVM. > > I found this sort of "informed guess" to be much more reliable than trying > to figure things out empirically by monitoring processes over time etc. ... > anyway, hope that helps, curious to know what you ultimately end up with. > > Cheers, > Tom > > On Thu, Aug 3, 2017 at 10:31 AM, Meg Figura <[email protected] > <javascript:>> wrote: > >> Hi Sebastian, >> >> Our product runs within the JVM, within a (Hadoop) YARN container. >> Similar to your situation, YARN will kill the container if it goes over the >> amount of memory reserved for the container. Java heap sizes (-Xmx) for the >> apps we run within containers vary from about 6GB to about 31GB, so this >> may be completely inappropriate if you use much smaller heaps, but here is >> the heuristic we use on Java 8. 'jvmMemory' is the -Xmx setting given to >> the JVM and adjustJvmMemoryForYarn() gives the size of the container we >> request. >> >> private static int getReservedCodeCacheSize(int jvmMemory) >> { >> return 100; >> } >> >> private static int getMaxMetaspaceSize(int jvmMemory) >> { >> return 256; >> } >> >> private static int getCompressedClassSpaceSize(int jvmMemory) >> { >> return 256; >> } >> >> private static int getExtraJvmOverhead(int jvmMemory) >> { >> if (jvmMemory <= 2048) >> { >> return 1024; >> } >> else if(jvmMemory <= (1024 * 16)) >> { >> return 2048; >> } >> else if(jvmMemory <= (1024 * 31)) >> { >> return 5120; >> } >> else >> { >> return 8192; >> } >> } >> >> public static int adjustJvmMemoryForYarn(int jvmMemory) >> { >> if (jvmMemory == 0) >> { >> return 0; >> } >> >> return jvmMemory + >> getReservedCodeCacheSize(jvmMemory) + >> getMaxMetaspaceSize(jvmMemory) + >> getCompressedClassSpaceSize(jvmMemory) + >> getExtraJvmOverhead(jvmMemory); >> } >> >> >> >> If the app uses any significant off-heap memory, we just add this to the >> container size. >> >> Obviously, this isn't optimal, but it does prevent the "OOM killer" from >> kicking in. I'm interested to see if anyone has a better solution! >> >> -Meg >> >> >> >> On Thursday, August 3, 2017 at 5:17:11 AM UTC-4, Sebastian Łaskawiec >> wrote: >>> >>> Hey, >>> >>> Before digging into the problem, let me say that I'm very happy to meet >>> you! My name is Sebastian Łaskawiec and I've been working for Red Hat >>> focusing mostly on in memory store solutions. A while ago I attended JVM >>> performance and profiling workshop lead by Martin, which was an incredible >>> experience to me. >>> >>> Over the last a couple of days I've been working on tuning and sizing >>> our app for Docker Containers. I'm especially interested in running JVM >>> without swap and constraining memory. Once you hit the memory limit, the >>> OOM Killer kicks and takes your application down. Rafael wrote pretty good >>> pragmatic description here [1]. >>> >>> I'm currently looking for some good practices for measuring and tuning >>> JVM memory size. I'm currently using: >>> >>> - The JVM native memory tracker [2] >>> - pmap -x, which gives me RSS >>> - jstat -gccause, which gives me an idea how GC is behaving >>> - dstat which is not CGroups aware but gives me an overall idea >>> about paging, CPU and memory >>> >>> Here's an example of a log that I'm analyzing [3]. Currently I'm trying >>> to adjust Xmx and Xms correctly so that my application fills the >>> constrained container but doesn't spill out (which would result in OOM Kill >>> done by the kernel). The biggest problem that I have is how to measure the >>> remaining amount of memory inside the container? Also I'm not sure why the >>> amount of committed JVM memory is different from RSS reported by pmap -x? >>> Could you please give me a hand with this? >>> >>> Thanks, >>> Sebastian >>> >>> [1] https://developers.redhat.com/blog/2017/03/14/java-inside-docker/ >>> [2] >>> https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html >>> [3] https://gist.github.com/slaskawi/a6ddb32e1396384d805528884f25ce4b >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "mechanical-sympathy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > *Tom Lee */ https://neeveresearch.com / @tglee <http://twitter.com/tglee> > > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
