On Fri, Nov 3, 2017 at 12:08 AM, Michael Vorburger <vorbur...@redhat.com> wrote:
> On Thu, Nov 2, 2017 at 11:59 PM, Jamo Luhrsen <jluhr...@gmail.com> wrote: > >> On 11/02/2017 03:49 PM, Michael Vorburger wrote: >> > On Thu, Nov 2, 2017 at 11:02 PM, Luis Gomez <ece...@gmail.com <mailto: >> ece...@gmail.com>> wrote: >> > >> > So from the time I was debugging the issue with Jamo and Thanh, it >> seems there is a memory leak but: >> > >> > 1) JVM does not kill itself, the OS does instead after the java >> process grows to 3.7G in a VM of 4G RAM (note Xmx is set >> > to 2G but still the jvm goes far beyond that). >> > >> > >> > duh! Weird. There are various online articles re. how a JVM can consume >> more than it's Xmx, mentioning total mem is some >> > magic combination of, AFAIK, Xmx + Threads + Stacks + PermGen/Metaspace >> + Native + JIT, so: >> > >> > Hopefully it's not JUST that you THINK Xmx is 2 GB but it's actually >> more.. say 4G, then on a small 4G VM, I could totally >> > see that happen; this is the scenario I described earlier, >> > on https://lists.opendaylight.org/pipermail/controller-dev/2 >> 017-October/014026.html >> > >> > It's not that I don't trust you guys, but you're like 250% super double >> extra sure sure Xmx is 2G, aren't you? ;) Maybe just >> > do a ps wwwwwww thing to see the full arg line and double check it. In >> the crash message shown >> > on https://jira.opendaylight.org/browse/NETVIRT-974 there are only >> environment variables, not the actual values. Would you >> > terribly mind doing that and sharing the actual full JVM start up line >> here, just so that we're sure? >> >> yeah, we (three of us) stared at the live VM and manually verified that >> it's running with 4G >> and Xmx is set to 2048M. we watched it hover at 1.6G-ish for a while, >> then in a matter of >> seconds it raced up past 3G and the OS killed it. >> >> The other clue is that this is not happening in the jobs running Carbon >> which used the same >> java opts, same VM image and size, etc. >> >> >> > PermGen/Metaspace is classes loaded; that's.. unlikely, IMHO. >> > >> > Theoretically I guess it could be some JNI crap Native memory usage >> within the JVM going crazy. Someone remind me what JNI >> > pieces we have in the puzzle? Karaf has some Console Shell thing that's >> native; but I'm guessing that's unlikely. Then there >> > is a native LevelDB, right? What if it's that which goes crazy? >> > >> > Or, quote: "Off-heap allocations. If you happen to use off-heap memory, >> for example while using direct or mapped ByteBuffers >> > yourself or via some clever 3rd party API then voila – you are >> extending your heap to something you actually cannot control >> > via JVM configuration." - anyone knows if we / CDS / controller / Akka >> does this kind of stuff? (AFAIK this includes "memory >> > mapped files")j >> > >> > Thread memory usage I don't think should just blew up like this, even >> if we did spawn threads like crazy in a bug (which >> > wouldn't surprise me), that would lead to an exception and more orderly >> termination, AFAIK. I could write a quick Java Test >> > app to confirm if this assumption is correct. >> > > quickly wrote ThreadExplosion.java just attached to NETVIRT-974 to > convince myself that's unlikely it (because that would "just" cause an > "java.lang.OutOfMemoryError: unable to create new native thread", but not > what you're seeing here).. so then my best bet is on JNI or "off heap" - > this is going to be fun to figure out. > > > 2) The issue happens so fast that we did not have time to take a >> memory dump with map. >> > >> > >> > what is map? We would need a HPROF produced by jmap; just a typo? >> >> yep, typo. jmap is what we were tying to use. We did grab a 1.5g dump in >> the beginning >> when we were verifying the jmap command but it was deleted when we did >> some other >> thing later. >> >> > BTW: Could we see a HPROF by jmap you take while it's still running, >> just before it goes crazy and gets killed? MAYBE with a >> > bit of luck, we already see something interesting in there.. >> >> I'll follow up with Thanh or Anil (has to be an LF person) when I have >> something >> running to do this. per your suggestion, we can use xz to compress it to >> hopefully >> make it easier to pass around. >> >> > So I wonder if there is some java memory combination of parameters >> to prevent OS to kill the JVM, something like if the >> > total java memory (Xmx and others)=3G, abort and generate heap dump. >> > >> > >> > No, I don't think so - as Jamo says, the JVM seems to just get kill-9'd >> by the OS, so no JVM parameter will helps us; it's... >> > like.. too late. But if we can't figure it out using any of the ideas >> above, then I guess we'll have to go lower.. I'm a >> > little bit less familiar with the OS level stuff, but I'm sure one of >> you know, or can figure out how, to get the OS to >> > produce one of those coredump files when it kills a process? Thing is I >> don't know what to do with such a file, and how to >> > analyze its memory content... but I guess we'll have to find someone >> who does! Once you have it. >> >> anyone know how to do this? >> > > this approach is probably the best next step here then... > by researching more, I just learnt a fun new trick ;) which could perhaps help us to learn what's going on here: would you mind to add the following magic incantations to the JVM options, e.g. by editing the bin/karaf script: DEFAULT_JAVA_OPTS="$DEFAULT_JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -Xcheck:jni -XX:NativeMemoryTracking=detail -XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics -XX:-AutoShutdownNMT" and then let it do its thing and then share what was on the JVM STDOUT ? Normally, this will cause it to print (a lot of) stuff on JVM exit. I'm not 100% sure if the OS OOM kind of kill will still give it chance to do that.. it depends on what "signal" that uses. In a quick local test, a kill -9 <PID> does not, but just a kill <PID> (= TERM) does... so it's worth a try. If there's nothing on exit, then the same output which above would produce at exit can be obtained during normal runtime via the following command, if you manage to run that some time (shortly) before it dies: jcmd <PID> VM.native_memory detail scale=MB > > >> Thanks, >> JamO >> >> >> > pmap of the JVM proc is mentioned as also being useful in some psots >> online >> > >> > >> > >> > BR/Luis >> > >> > >> > > On Nov 2, 2017, at 2:53 PM, Jamo Luhrsen <jluhr...@gmail.com >> <mailto:jluhr...@gmail.com>> wrote: >> > > >> > > >> > > >> > > On 11/02/2017 02:02 PM, Michael Vorburger wrote: >> > >> On Thu, Nov 2, 2017 at 9:32 PM, Jamo Luhrsen <jluhr...@gmail.com >> <mailto:jluhr...@gmail.com> >> > <mailto:jluhr...@gmail.com <mailto:jluhr...@gmail.com>>> wrote: >> > >> >> > >> +integration-dev >> > >> >> > >> nitrogen SR1 blocker bug for this problem: >> > >> >> > >> https://jira.opendaylight.org/browse/NETVIRT-974 < >> https://jira.opendaylight.org/browse/NETVIRT-974> >> > <https://jira.opendaylight.org/browse/NETVIRT-974 < >> https://jira.opendaylight.org/browse/NETVIRT-974>> >> > >> >> > >> I'm actively debugging in the sandbox, although it's a very >> heavy process >> > >> (many hours per iteration). >> > >> >> > >> wondering if there are any extra options we can pass to the >> java process >> > >> that might shed more light. This is a very fast and silent >> death. >> > >> >> > >> >> > >> sounds like a memory leak.. we should have a hs_err_pid*.log >> file and *have* to have an *.hprof file to know where and >> > find a >> > >> fix for an OOM... >> > >>> have you been able to re-double-check if these files are't >> already produced somewhere? How about just doing a dumb: >> > >> >> > >> sudo find / -name "hs_err_pid*.log" >> > >> sudo find / -name "*.hprof" >> > >> >> > >> the hprof should be produced by I can see that in $ODL/bin/karaf >> we already have "-XX:+HeapDumpOnOutOfMemoryError" >> > >> on DEFAULT_JAVA_OPTS... >> > > >> > > >> > > yes, we do this HeapdumpOnOutOfMemoryError. >> > > >> > >> to fix the folder where it would write the HPROF into, you could >> add: -XX:HeapDumpPath=/a/folder/you/can/recover >> > >> >> > >> can't wait to get my hands on a hs_err_pid*.log & *.hprof from >> this... ;=) >> > > >> > > >> > > no there is no hs_err* or *hprof here. The OS is killing the PID >> because it's >> > > consuming too much memory. I don't think the OS even cares that >> this is java. >> > > Same as me doing a kill -9, I presume. >> > > >> > > :( >> > > >> > > JamO >> > > >> > > >> > >> JamO >> > >> >> > >> >> > >> >> > >> >> > >> On 10/31/2017 06:11 PM, Sam Hague wrote: >> > >>> >> > >>> >> > >>> On Tue, Oct 31, 2017 at 6:44 PM, Anil Vishnoi < >> vishnoia...@gmail.com <mailto:vishnoia...@gmail.com> >> > <mailto:vishnoia...@gmail.com <mailto:vishnoia...@gmail.com>> >> <mailto:vishnoia...@gmail.com >> > <mailto:vishnoia...@gmail.com> <mailto:vishnoia...@gmail.com >> <mailto:vishnoia...@gmail.com>>>> wrote: >> > >>> >> > >>> is it possible to collect dmesg output? That can give an >> idea if it's a JVM native OOM. >> > >>> >> > >>> Yes, we already collect all those for the openstack nodes, so >> we just need to include it for the ODL node. >> > >>> >> > >>> >> > >>> On Tue, Oct 31, 2017 at 3:40 PM, Michael Vorburger < >> vorbur...@redhat.com <mailto:vorbur...@redhat.com> >> > <mailto:vorbur...@redhat.com <mailto:vorbur...@redhat.com>> >> > >> <mailto:vorbur...@redhat.com <mailto:vorbur...@redhat.com> >> <mailto:vorbur...@redhat.com >> > <mailto:vorbur...@redhat.com>>>> wrote: >> > >>> >> > >>> On Tue, Oct 31, 2017 at 11:02 PM, Jamo Luhrsen < >> jluhr...@gmail.com <mailto:jluhr...@gmail.com> >> > <mailto:jluhr...@gmail.com <mailto:jluhr...@gmail.com>> <mailto: >> jluhr...@gmail.com <mailto:jluhr...@gmail.com> >> > <mailto:jluhr...@gmail.com <mailto:jluhr...@gmail.com>>>> wrote: >> > >>> >> > >>> On 10/31/2017 12:22 AM, Michael Vorburger wrote: >> > >>> > On Tue, Oct 31, 2017 at 12:44 AM, Jamo Luhrsen < >> jluhr...@gmail.com <mailto:jluhr...@gmail.com> >> > <mailto:jluhr...@gmail.com <mailto:jluhr...@gmail.com>> <mailto: >> jluhr...@gmail.com <mailto:jluhr...@gmail.com> >> > <mailto:jluhr...@gmail.com <mailto:jluhr...@gmail.com>>> >> > >> <mailto:jluhr...@gmail.com <mailto:jluhr...@gmail.com> >> <mailto:jluhr...@gmail.com <mailto:jluhr...@gmail.com>> >> > <mailto:jluhr...@gmail.com <mailto:jluhr...@gmail.com> <mailto: >> jluhr...@gmail.com <mailto:jluhr...@gmail.com>>>>> wrote: >> > >>> > >> > >>> > On 10/30/2017 01:29 PM, Tom Pantelis wrote: >> > >>> > > On Mon, Oct 30, 2017 at 4:25 PM, Sam Hague >> <sha...@redhat.com <mailto:sha...@redhat.com> >> > <mailto:sha...@redhat.com <mailto:sha...@redhat.com>> <mailto: >> sha...@redhat.com <mailto:sha...@redhat.com> >> > <mailto:sha...@redhat.com <mailto:sha...@redhat.com>>> >> > >>> <mailto:sha...@redhat.com <mailto: >> sha...@redhat.com> <mailto:sha...@redhat.com >> > <mailto:sha...@redhat.com>> <mailto:sha...@redhat.com <mailto: >> sha...@redhat.com> <mailto:sha...@redhat.com >> > <mailto:sha...@redhat.com>>>> >> > >> <mailto:sha...@redhat.com <mailto:sha...@redhat.com> <mailto: >> sha...@redhat.com <mailto:sha...@redhat.com>> >> > <mailto:sha...@redhat.com <mailto:sha...@redhat.com> <mailto: >> sha...@redhat.com <mailto:sha...@redhat.com>>> >> > >>> > <mailto:sha...@redhat.com <mailto: >> sha...@redhat.com> <mailto:sha...@redhat.com >> > <mailto:sha...@redhat.com>> <mailto:sha...@redhat.com <mailto: >> sha...@redhat.com> <mailto:sha...@redhat.com >> > <mailto:sha...@redhat.com>>>>>> wrote: >> > >>> > > On Mon, Oct 30, 2017 at 3:02 PM, Tom >> Pantelis <tompante...@gmail.com >> > <mailto:tompante...@gmail.com> <mailto:tompante...@gmail.com >> <mailto:tompante...@gmail.com>> >> > <mailto:tompante...@gmail.com <mailto:tompante...@gmail.com> >> <mailto:tompante...@gmail.com <mailto:tompante...@gmail.com>>> >> > >> <mailto:tompante...@gmail.com <mailto:tompante...@gmail.com> >> <mailto:tompante...@gmail.com >> > <mailto:tompante...@gmail.com>> <mailto:tompante...@gmail.com >> <mailto:tompante...@gmail.com> >> > <mailto:tompante...@gmail.com <mailto:tompante...@gmail.com>>>> >> > >>> > <mailto:tompante...@gmail.com <mailto: >> tompante...@gmail.com> <mailto:tompante...@gmail.com >> > <mailto:tompante...@gmail.com>> <mailto:tompante...@gmail.com >> <mailto:tompante...@gmail.com> >> > <mailto:tompante...@gmail.com <mailto:tompante...@gmail.com>>> >> > >> <mailto:tompante...@gmail.com <mailto:tompante...@gmail.com> >> <mailto:tompante...@gmail.com >> > <mailto:tompante...@gmail.com>> >> > >>> <mailto:tompante...@gmail.com <mailto: >> tompante...@gmail.com> <mailto:tompante...@gmail.com >> > <mailto:tompante...@gmail.com>>>>>> wrote: >> > >>> > > On Mon, Oct 30, 2017 at 2:49 PM, >> Michael Vorburger <vorbur...@redhat.com >> > <mailto:vorbur...@redhat.com> <mailto:vorbur...@redhat.com <mailto: >> vorbur...@redhat.com>> <mailto:vorbur...@redhat.com >> > <mailto:vorbur...@redhat.com> <mailto:vorbur...@redhat.com <mailto: >> vorbur...@redhat.com>>> >> > >> <mailto:vorbur...@redhat.com <mailto:vorbur...@redhat.com> >> <mailto:vorbur...@redhat.com >> > <mailto:vorbur...@redhat.com>> <mailto:vorbur...@redhat.com >> <mailto:vorbur...@redhat.com> <mailto:vorbur...@redhat.com >> > <mailto:vorbur...@redhat.com>>>> >> > >>> <mailto:vorbur...@redhat.com <mailto: >> vorbur...@redhat.com> <mailto:vorbur...@redhat.com >> > <mailto:vorbur...@redhat.com>> <mailto:vorbur...@redhat.com >> <mailto:vorbur...@redhat.com> <mailto:vorbur...@redhat.com >> > <mailto:vorbur...@redhat.com>>> >> > >> <mailto:vorbur...@redhat.com <mailto:vorbur...@redhat.com> >> <mailto:vorbur...@redhat.com <mailto:vorbur...@redhat.com>> >> > >>> <mailto:vorbur...@redhat.com <mailto: >> vorbur...@redhat.com> <mailto:vorbur...@redhat.com >> > <mailto:vorbur...@redhat.com>>>>>> wrote: >> > >>> > > >> > >>> > > Hi Sam, >> > >>> > > >> > >>> > > On Mon, Oct 30, 2017 at 7:45 >> PM, Sam Hague <sha...@redhat.com >> > <mailto:sha...@redhat.com> <mailto:sha...@redhat.com <mailto: >> sha...@redhat.com>> <mailto:sha...@redhat.com >> > <mailto:sha...@redhat.com> <mailto:sha...@redhat.com <mailto: >> sha...@redhat.com>>> >> > >>> <mailto:sha...@redhat.com <mailto: >> sha...@redhat.com> <mailto:sha...@redhat.com >> > <mailto:sha...@redhat.com>> <mailto:sha...@redhat.com <mailto: >> sha...@redhat.com> <mailto:sha...@redhat.com >> > <mailto:sha...@redhat.com>>>> >> > >> <mailto:sha...@redhat.com <mailto:sha...@redhat.com> <mailto: >> sha...@redhat.com <mailto:sha...@redhat.com>> >> > <mailto:sha...@redhat.com <mailto:sha...@redhat.com> <mailto: >> sha...@redhat.com <mailto:sha...@redhat.com>>> >> > >>> <mailto:sha...@redhat.com <mailto: >> sha...@redhat.com> <mailto:sha...@redhat.com >> > <mailto:sha...@redhat.com>> <mailto:sha...@redhat.com <mailto: >> sha...@redhat.com> >> > >> <mailto:sha...@redhat.com <mailto:sha...@redhat.com>>>>>> >> wrote: >> > >>> > > >> > >>> > > Stephen, Michael, Tom, >> > >>> > > >> > >>> > > do you have any ways to >> collect debugs when ODL crashes in CSIT? >> > >>> > > >> > >>> > > >> > >>> > > JVMs (almost) never "just >> crash" without a word... either some code >> > >>> does java.lang.System.exit(), which you may >> > >>> > > remember we do in the CDS/Akka >> code somewhere, or there's a bug in the JVM >> > implementation - >> > >>> in which case there >> > >>> > > should be a one of those JVM >> crash logs type things - a file named something >> > >>> like hs_err_pid22607.log in the >> > >>> > > "current working" directory. >> Where would that be on these CSIT runs, and are the >> > CSIT JJB >> > >>> jobs set up to preserve >> > >>> > > such JVM crash log files and >> copy them over to logs.opendaylight.org >> > <http://logs.opendaylight.org> >> > >> <http://logs.opendaylight.org> >> > >>> <http://logs.opendaylight.org> < >> http://logs.opendaylight.org> >> > >>> > <http://logs.opendaylight.org> ? >> > >>> > > >> > >>> > > >> > >>> > > Akka will do System.exit() if it >> encounters an error serious for that. But it doesn't do it >> > >>> silently. However I >> > >>> > > believe we disabled the automatic >> exiting in akka. >> > >>> > > >> > >>> > > Should there be any logs in ODL for >> this? There is nothing in the karaf log when this >> > happens. It >> > >>> literally just stops. >> > >>> > > >> > >>> > > The karaf.console log does say the >> karaf process was killed: >> > >>> > > >> > >>> > > /tmp/karaf-0.7.1-SNAPSHOT/bin/karaf: >> line 422: 11528 Killed ${KARAF_EXEC} "${JAVA}" ${JAVA_OPTS} >> > >>> "$NON_BLOCKING_PRNG" >> > >>> > > >> > -Djava.endorsed.dirs="${JAVA_ENDORSED_DIRS}" >> -Djava.ext.dirs="${JAVA_EXT_DIRS}" >> > >>> > > >> > -Dkaraf.instances="${KARAF_HOME}/instances" >> -Dkaraf.home="${KARAF_HOME}" >> > >> -Dkaraf.base="${KARAF_BASE}" >> > >>> > > -Dkaraf.data="${KARAF_DATA}" >> -Dkaraf.etc="${KARAF_ETC}" -Dkaraf.restart.jvm.supported=true >> > >>> > > -Djava.io.tmpdir="${KARAF_DATA}/tmp" >> > >>> -Djava.util.logging.config.fil >> e="${KARAF_BASE}/etc/java.util.logging.properties" >> > >>> > > ${KARAF_SYSTEM_OPTS} ${KARAF_OPTS} >> ${OPTS} "$@" -classpath "${CLASSPATH}" ${MAIN} >> > >>> > > >> > >>> > > In the CSIT robot files we can see the >> below connection errors so ODL is not responding to new >> > >>> requests. This plus the >> > >>> > > above lead to think ODL just died. >> > >>> > > >> > >>> > > [ WARN ] Retrying (Retry(total=2, >> connect=None, read=None, redirect=None, status=None)) after >> > >>> connection broken by >> > >>> > > 'NewConnectionError('<request >> s.packages.urllib3.connection.HTTPConnection object at 0x5ca2d50>: >> > >>> Failed to establish a new >> > >>> > > connection: [Errno 111] Connection >> refused',)' >> > >>> > > >> > >>> > > >> > >>> > > >> > >>> > > That would seem to indicate something did >> a kill -9. As Michael said, if the JVM crashed there >> > >> would be >> > >>> an hs_err_pid file >> > >>> > > and it would log a message about it >> > >>> > >> > >>> > yeah, this is where my money is at as well. >> The OS must be dumping it because it's >> > >>> > misbehaving. I'll try to hack the job to >> start collecting os level log info (e.g. journalctl, etc) >> > >>> > >> > >>> > >> > >>> > JamO, do make sure you collect not just OS level >> but also the JVM's hs_err_*.log file (if any); my >> > bet is a >> > >>> JVM more than an >> > >>> > OS level crash... >> > >>> >> > >>> where are these hs_err_*.log files going to be? >> > >>> >> > >>> >> > >>> they would be in the "current working directory", like >> what was the "pwd" when the JVM was started.. >> > >>> >> > >>> >> > >>> This is such a dragged out process to debug. These >> > >>> jobs take 3+ hours and our problem only comes >> sporadically. ...sigh... >> > >>> >> > >>> But, good news is that I think we've confirmed >> it's an oom. but an OOM from the OS perspective, >> > >>> if I'm not mistaken. >> > >>> >> > >>> >> > >>> OK that kind of thing could happen if you ran an ODL >> JVM in this kind of situation: >> > >>> >> > >>> * VM with say 4 GB of RAM, and no swap >> > >>> * JVM like ODL starts with Xms 1 GB and Xmx 2 GB, so >> reserves 1 and plans expand to 2, when needed >> > >>> * other stuff eats up remaining e.g. 3 GB >> > >>> * JVM wants to expand, asks OS for 1 GB, but there is >> none left - so boum >> > >>> >> > >>> but AFAIK (I'm not 100% sure) there would still be one >> of those hs_err_*.log files with some details confirming >> > >> above >> > >>> (like "out of native memory", kind of thing). >> > >>> >> > >>> >> > >>> here's what I saw in a sandbox job [a] that just >> hit this: >> > >>> >> > >>> Out of memory: Kill process 11546 (java) score 933 >> or sacrifice child >> > >>> (more debug output is there in the console log) >> > >>> >> > >>> These ODL systems start with 4G and we are setting >> the max mem for the odl java >> > >>> process to be 2G. >> > >>> >> > >>> >> > >>> erm, I'm not quite following what is 2 and what is 3 >> here.. but does my description above help you to narrow >> > >> this down? >> > >>> >> > >>> >> > >>> I don't think we see this with Carbon, which makes >> me believe it's *not* some problem from outside >> > >>> of ODL (e.g. not a kernel bug from when we updated >> the java builder image back on 10/20) >> > >>> >> > >>> I'll keep digging at this. Ideas are welcome for >> things to look at. >> > >>> >> > >>> >> > >>> >> > >>> [a] >> > >>> >> > >> >> > https://jenkins.opendaylight.org/sandbox/job/ >> netvirt-csit-1node-openstack-pike-jamo-upstream-stateful- >> snat-conntrack-oxygen/7/consoleFull >> > <https://jenkins.opendaylight.org/sandbox/job/ >> netvirt-csit-1node-openstack-pike-jamo-upstream-stateful- >> snat-conntrack-oxygen/7/consoleFull> >> > >> >> > <https://jenkins.opendaylight.org/sandbox/job/ >> netvirt-csit-1node-openstack-pike-jamo-upstream-stateful- >> snat-conntrack-oxygen/7/consoleFull >> > <https://jenkins.opendaylight.org/sandbox/job/ >> netvirt-csit-1node-openstack-pike-jamo-upstream-stateful- >> snat-conntrack-oxygen/7/consoleFull>> >> > >>> >> > >> >> > <https://jenkins.opendaylight.org/sandbox/job/netvirt- >> csit-1node-openstack-pike-jamo-upstream-stateful-snat- >> conntrack-oxygen/7/consoleFull >> > <https://jenkins.opendaylight.org/sandbox/job/ >> netvirt-csit-1node-openstack-pike-jamo-upstream-stateful- >> snat-conntrack-oxygen/7/consoleFull> >> > >> >> > <https://jenkins.opendaylight.org/sandbox/job/ >> netvirt-csit-1node-openstack-pike-jamo-upstream-stateful- >> snat-conntrack-oxygen/7/consoleFull >> > <https://jenkins.opendaylight.org/sandbox/job/ >> netvirt-csit-1node-openstack-pike-jamo-upstream-stateful- >> snat-conntrack-oxygen/7/consoleFull>>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> > BTW: The most common fix ;) for JVM crashes >> often is simply upgrading to the latest available patch >> > >> version of OpenJDK.. but >> > >>> > I'm guessing/hoping we run from RPM and already >> have the latest - or is this possibly running on an older >> > >> JVM version package >> > >>> > that was somehow "held back" via special dnf >> instructions, or manually installed from a ZIP, kind of >> > thing? >> > >>> >> > >>> >> > >>> these systems are built and updated periodically. >> jdk is installed with "yum install". The specific version >> > >>> in [a] is: >> > >>> >> > >>> 10:57:33 Set Java version >> > >>> 10:57:34 JDK default version... >> > >>> 10:57:34 openjdk version "1.8.0_144" >> > >>> 10:57:34 OpenJDK Runtime Environment (build >> 1.8.0_144-b01) >> > >>> 10:57:34 OpenJDK 64-Bit Server VM (build >> 25.144-b01, mixed mode) >> > >>> >> > >>> >> > >>> OK, that seems to be the latest one I also have >> locally on Fedora 26. >> > >>> >> > >>> Thanks, >> > >>> JamO >> > >>> >> > >>> >> > >>> >> > >>> > JamO >> > >>> > >> > >>> > >> > >>> > > >> > >>> > > ______________________________ >> _________________ >> > >>> > > controller-dev mailing list >> > >>> > > controller-dev@lists.opendaylight.org >> <mailto:controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>> >> > >> <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>>> >> > >>> <mailto:controller-dev@lists.opendaylight.org >> <mailto:controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>> >> > >> <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>>>> >> > >>> > > https://lists.opendaylight.org >> /mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>> >> > >>> <https://lists.opendaylight.or >> g/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>>> >> > >>> > <https://lists.opendaylight.o >> rg/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>> >> > >>> <https://lists.opendaylight.or >> g/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>>>> >> > >>> > > >> > >>> > _____________________________ >> __________________ >> > >>> > controller-dev mailing list >> > >>> > controller-dev@lists.opendaylight.org >> <mailto:controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>> >> > >> <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>>> >> > >>> <mailto:controller-dev@lists.opendaylight.org >> <mailto:controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>> >> > >> <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>>>> >> > >>> > https://lists.opendaylight.or >> g/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>> >> > >>> <https://lists.opendaylight.or >> g/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>>> >> > >>> > <https://lists.opendaylight.o >> rg/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>> >> > >>> <https://lists.opendaylight.or >> g/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>>>> >> > >>> > >> > >>> > >> > >>> >> > >>> >> > >>> >> > >>> _______________________________________________ >> > >>> controller-dev mailing list >> > >>> controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>> >> > >> <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>>> >> > >>> https://lists.opendaylight.org >> /mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>> >> > >>> <https://lists.opendaylight.or >> g/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> -- >> > >>> Thanks >> > >>> Anil >> > >>> >> > >>> _______________________________________________ >> > >>> controller-dev mailing list >> > >>> controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>> >> > >> <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>>> >> > >>> https://lists.opendaylight.org >> /mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>> >> > >>> <https://lists.opendaylight.or >> g/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> _______________________________________________ >> > >>> controller-dev mailing list >> > >>> controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>> >> > >>> https://lists.opendaylight.org/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>> >> > >>> >> > >> _______________________________________________ >> > >> controller-dev mailing list >> > >> controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > <mailto:controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org>> >> > >> https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> <https://lists.opendaylight.org/mailman/listinfo/controller- >> dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>> >> > >> >> > >> >> > > _______________________________________________ >> > > controller-dev mailing list >> > > controller-dev@lists.opendaylight.org <mailto: >> controller-dev@lists.opendaylight.org> >> > > https://lists.opendaylight.org/mailman/listinfo/controller-dev >> > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> >> > >> > >> > >
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
