Well, I think Thanh's point is that if the output.xml is too big, we just add more RAM to fix it, although instead we could add some swap which is "slower RAM"
JamO On 11/29/2017 09:13 AM, Luis Gomez wrote: > I do not remember all the issues related with memory but at least swap is > required in the robot VM to generate the log.html > when output.xml file is big. > >> On Nov 29, 2017, at 2:56 AM, Thanh Ha <[email protected] >> <mailto:[email protected]>> wrote: >> >> We can certainly add swap but imo swap is just a slower ram. At least in my >> experience when a system starts needing to swap >> things have already gone beyond recovery and swap just delays the inevitable >> program crash. In any case there's a few ways >> we can add swap if we really want to: >> >> 1) At the job level, create a swap file and activate it for the size needed >> for the job. >> 2) At the jenkins-init level (via jenkins-scripts in releng/builder) we can >> add swap for systems here >> 3) Bake it into the VM via packer file >> >> If we want to try this with a specific job type then I would try it with >> option 1 first and see how things go. If we find >> it really useful we can move it into 2 or 3. >> >> Regards, >> Thanh >> >> On Mon, Nov 13, 2017 at 6:19 PM, Luis Gomez <[email protected] >> <mailto:[email protected]>> wrote: >> >> +1, adding Andy: is it possible to get swap space in the VM from the >> cloud provider? otherwise is it possible to modify >> the releng/builder packer scripts to add the swap? >> >>> On Nov 13, 2017, at 3:05 PM, Sam Hague <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> On Thu, Nov 9, 2017 at 5:45 AM, Stephen Kitt <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Exactly, that’s what I was alluding to with “the lack of swap >>> probably >>> doesn’t help” ;-). We should really configure VMs with a small >>> amount >>> of swap, it can save the kernel in tricky situations... >>> >>> I was looking at this also and wondering about swap. We had a different >>> issue where the robot VMs blow up on producing >>> the log.html file because it ends up being a 1gb file. With swap I >>> think it would have passed fine. As is, we bumped >>> the 2gb vm to a 4gb to make it work. So we will likely hit it again. >>> >>> >>> On Thu, 9 Nov 2017 01:47:08 -0800 >>> Anil Vishnoi <[email protected] <mailto:[email protected]>> >>> wrote: >>> >>> > I suspect you might hit it again if you run it a bit longer >>> because >>> > of this >>> > >>> > [Thu Nov 2 05:58:08 2017] Free swap = 0kB >>> > [Thu Nov 2 05:58:08 2017] Total swap = 0kB >>> > >>> > >>> > On Thu, Nov 9, 2017 at 1:39 AM, Stephen Kitt <[email protected] >>> <mailto:[email protected]>> wrote: >>> > >>> > > On Thu, 9 Nov 2017 10:28:14 +0100 >>> > > Robert Varga <[email protected] <mailto:[email protected]>> wrote: >>> > > > On 02/11/17 23:02, Luis Gomez wrote: >>> > > > > 1) JVM does not kill itself, the OS does instead after the >>> java >>> > > > > process grows to 3.7G in a VM of 4G RAM (note Xmx is set >>> to 2G >>> > > > > but still the jvm goes far beyond that). >>> > > > >>> > > > Indicates this lies out side of heap -- check thread count. >>> > > >>> > > We verified separately that this is an OOM issue, but one >>> detected >>> > > by the kernel rather than by the JVM (the OOM killer kills the >>> JVM, >>> > > see >>> > > >>> https://jira.opendaylight.org/secure/attachment/14207/dmesg.log.txt >>> >>> <https://jira.opendaylight.org/secure/attachment/14207/dmesg.log.txt> >>> > > for details; the number of threads wasn’t an issue here, but the >>> > > lack of swap probably didn’t help). >>> > > >>> > > Upgrading to OpenJDK 8 patch 151 fixed the problem, it might >>> have >>> > > been related to one of the several memory usage bugs in 144 that >>> > > were fixed in 151. It’s probably just moving the goalposts >>> though >>> > > since the problem was new — basically, I suspect we recently >>> > > started using a little too much off-heap memory for some reason, >>> > > and the upgrade to 151 reduces the JVM’s memory usage enough to >>> > > make us fit in our VMs again. >>> > > >>> > > Regards, >>> > > >>> > > Stephen >> >> > > > > _______________________________________________ > integration-dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/integration-dev > _______________________________________________ controller-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/controller-dev
