Reading through the mail thread, I doubt that OS is killing the JVM, it's definitely a native OOM, where JVM is trying to do malloc/calloc and it does not find the memory and malloc crashes the JVM process(seg fault). I have seen many native OOM issue with 32 bit JVM because of it's 4gig virtual address space, but in 64 bit JVM, i have seen this issue when swap space is not enabled in your system. So the first check i would suggest is to check if your VM has swap space, if not, then i would suggest to create swap space for your VM and run JVM. This most probably will fix the issue, although your JVM performance will suck as OS will do lot of context switching for the JVM process. It might not be a native memory leak, it's might be that your JVM needs more memory to run. Now the question is why it's not failing in carbon. This can happen because of few reasons (1) Karaf 4 is loading more classes compared to karaf 3, so that will require more native memory to store those bytecodes. (2) Karaf 4 code is getting JITted faster compared to karaf3 (depends on the code execution stack), and that also will require more memory or (3) it's really some leak (most probably classloaded, nio leak)
Generally when native OOM happens you can verify it through two ways (1) check dmesg and see if it has OOM related message (2) You see that OS init dumps the coredump for that JVM process. If you are not seeing the core dump it means your ulimit -c is set to 0. I would suggest to look at the dmesg log first and see if there is anything like "Out of memory: Killed process XXXXX, UID YY, (java)", that will verify that it's native OOM. Now to debug it, you need a coredump. Check if your ulimit -c setting is set to unlimited, if not fire this command (ulimit -c unlimited) and then start the JVM and recreate the issue. This should generate the core dump. Once you have the core dump, attach the gdb to coredump and dump the current thread stack, this will tell you why JVM crashed. Most probably the top function of the stack trace will be calloc() or malloc(). Mostly i have seen the crashed thread is going to be native GC thread, NIO thread, File IO thread, classloader thread or native JNI thread. If the thread was doing classloader or jni activity, it is probably a sign of something wrong with the nitrogen controller code, but if it's anything else it might be that your JVM need more native memory to run karaf4 controller. On Mon, Nov 6, 2017 at 11:32 AM, Jamo Luhrsen <[email protected]> wrote: > > > On 11/03/2017 10:24 AM, Michael Vorburger wrote: > > On Fri, Nov 3, 2017 at 11:02 AM, Stephen Kitt <[email protected] <mailto: > [email protected]>> wrote: > > > > On Fri, 3 Nov 2017 10:56:48 +0100 > > Stephen Kitt <[email protected] <mailto:[email protected]>> wrote: > > > Another avenue could be to look at the overcommit settings, but I > > > suspect that the JVM would fail to start altogether if we adjusted > > > those. > > > > Actually that’s worth trying anyway: > > > > sysctl -w vm.overcommit_memory=2 > > > > before starting the JVM, will prevent it from over-allocating memory. > > We’ll get a heap dump when that happens, if the JVM actually manages > to > > start (which I suspect it won’t, with a 2G Xmx on a 4G system). > > > > > > > > https://jira.opendaylight.org/browse/NETVIRT-974 now has some sub-tasks > for various next actions.. > > I've updated NETVIRT-975 [0]. The TL;DR is that with a different host > image we do not > see the crash. > > I'm looking for opinions on the next steps, one of which could be that we > update > our images, close the bug and forget this ever happened :) > > JamO > > [0] > https://jira.opendaylight.org/browse/NETVIRT-974? > focusedCommentId=59941&page=com.atlassian.jira.plugin. > system.issuetabpanels:comment-tabpanel#comment-59941 > _______________________________________________ > controller-dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/controller-dev > -- Thanks Anil
_______________________________________________ controller-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/controller-dev
