Re: vmd losing VMs

2018-10-23 Thread Greg Steuck
Not that I expected anything to change considering the patches submitted, but Oct 21 snapshot (minus "Add support to create and convert disk images from existing images") is similarly afflicted. Any candidate fixes or patches for added logging will be put to test in short order :) ci-openbsd$

Re: vmd losing VMs

2018-10-17 Thread Dmitry Vyukov
; > the point of running sshd. At least that's what I surmised from the >> > posted >> > manager.log file. >> > https://gist.github.com/blackgnezdo/a69e83c42c0c4cbbd53c7f3b35e91632 >> >> >> This should be fine in the sense that it should not lead to vm

Re: vmd losing VMs

2018-10-17 Thread Greg Steuck
> Can you net out what the current issues you're facing? I can't seem to grok > the log file posted above (not sure what I'm looking at). From reviewing the > thread, it seems the core dumps are gone but there may (?) be some issues > still? The core dumps are gone indeed. Thanks for the prompt

Re: vmd losing VMs

2018-10-17 Thread Greg Steuck
> > manager.log file. > > https://gist.github.com/blackgnezdo/a69e83c42c0c4cbbd53c7f3b35e91632 > > > This should be fine in the sense that it should not lead to vmd losing VMs. I agree that that vmd shouldn't be losing the VMs in such a case. But some evidence shows they bec

Re: vmd losing VMs

2018-10-17 Thread Dmitry Vyukov
/blackgnezdo/a69e83c42c0c4cbbd53c7f3b35e91632 This should be fine in the sense that it should not lead to vmd losing VMs. If test machine does not come up live within some time frame, we kill it and create a new one. > Here's an excerpt: > 2018/10/16 19:35:08 VMs 2, executed 68322, cover 18582, crashes 78, repro 0 >

Re: vmd losing VMs

2018-10-16 Thread Greg Steuck
I think I see some evidence of occasional VM just never finishing booting to the point of running sshd. At least that's what I surmised from the posted manager.log file. https://gist.github.com/blackgnezdo/a69e83c42c0c4cbbd53c7f3b35e91632 Here's an excerpt: 2018/10/16 19:35:08 VMs 2, executed

Re: vmd losing VMs

2018-10-14 Thread Greg Steuck
Now that I'm running OpenBSD 6.4 (GENERIC.MP) #362: Thu Oct 11 04:53:41 MDT 2018, I can start debugging again. I just observed an interesting tidbit which I failed to notice before. Namely, there are also hanging vmctl processes trying to stop those spinning VMs. So, I tried to reproduce this

Re: vmd losing VMs

2018-10-10 Thread Dmitry Vyukov
On Tue, Oct 2, 2018 at 8:02 PM, Greg Steuck wrote: > Dmitry, is there an easy way get at the VMs output? You mean the test VM instances (vmd), right? We capture vmm/kernel output for crashes, you can see it in the Log columns for each crash here: https://syzkaller.appspot.com/#openbsd Also if

Re: vmd losing VMs

2018-10-09 Thread Greg Steuck
All my vmds are stuck after syzkaller was running for some 5 days (and found a couple of bugs!). I'll reinstall the system to get a fresh baseline. > 1. Are you getting any vmd cores? > * sysctl kern.nosuidcoredump=3 && mkdir /var/crash/vmd > * kill all vmd and restart the test. Not sure if

Re: vmd losing VMs

2018-10-02 Thread Mike Larkin
On Tue, Oct 02, 2018 at 11:02:57AM -0700, Greg Steuck wrote: > Dmitry, is there an easy way get at the VMs output? > > Here's dmesg from VMM_DEBUG-enabled kernel (built at "Add support for > RT3290 chipset by James Hastings."). No lockup thus far. > Yeah the output below shows normal behaviour.

Re: vmd losing VMs

2018-10-02 Thread Greg Steuck
Dmitry, is there an easy way get at the VMs output? Here's dmesg from VMM_DEBUG-enabled kernel (built at "Add support for RT3290 chipset by James Hastings."). No lockup thus far. OpenBSD 6.4 (VMM_DEBUG) #0: Tue Oct 2 10:37:13 PDT 2018 syzkaller@ci-openbsd.syzkaller

Re: vmd losing VMs

2018-10-02 Thread Reyk Floeter
Ok thanks. I wonder if there is a tool that can log everything from the VM's console ... It would be useful to see what happens to the VM before it goes into zombie mode. Reyk On Tue, Oct 02, 2018 at 10:33:11AM -0700, Greg Steuck wrote: > I believe I was unable to ssh into them or get cu to

Re: vmd losing VMs

2018-10-02 Thread Greg Steuck
I believe I was unable to ssh into them or get cu to elicit any characters. I'll verify next time it happens. On Tue, Oct 2, 2018 at 10:20 AM Reyk Floeter wrote: > On Tue, Oct 02, 2018 at 10:10:41AM -0700, Greg Steuck wrote: > > Naturally, bugs don't solve themselves :) Here's a log, it's not

Re: vmd losing VMs

2018-10-02 Thread Reyk Floeter
On Tue, Oct 02, 2018 at 10:10:41AM -0700, Greg Steuck wrote: > Naturally, bugs don't solve themselves :) Here's a log, it's not very > useful due to the lack of debugging symbols. Notice, that runaway vmds > don't die on their own, they just spin out of control. I'll do VMM_DEBUG > next. > "they

vmd losing VMs

2018-10-02 Thread Greg Steuck
Naturally, bugs don't solve themselves :) Here's a log, it's not very useful due to the lack of debugging symbols. Notice, that runaway vmds don't die on their own, they just spin out of control. I'll do VMM_DEBUG next. ci-openbsd# sysctl kern.nosuidcoredump=3 && mkdir /var/crash/vmd

Re: vmd losing VMs

2018-10-02 Thread Mike Larkin
On Mon, Oct 01, 2018 at 10:16:24PM -0700, Greg Steuck wrote: > Thanks Mike. > > I've upgraded from Sep 27th to Sep 29th snapshot and so far I haven't seen > the problem with: > > OpenBSD 6.4-beta (GENERIC.MP) #336: Sat Sep 29 08:13:15 MDT 2018 >

Re: vmd losing VMs

2018-10-01 Thread Greg Steuck
Thanks Mike. I've upgraded from Sep 27th to Sep 29th snapshot and so far I haven't seen the problem with: OpenBSD 6.4-beta (GENERIC.MP) #336: Sat Sep 29 08:13:15 MDT 2018 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP pd@ is saying he's not responsible for the fix, but

Re: vmd losing VMs

2018-09-30 Thread Mike Larkin
On Fri, Sep 28, 2018 at 05:01:27PM -0700, Greg Steuck wrote: > I've been running syzkaller for about a day now. It launches/kills VMs all > the time. Somewhere along the way vmd seems to have lost track of one of > its VMs. Notice how syzkaller ci-openbsd-main-1 is conspicuously missing > from

Re: vmd losing VMs

2018-09-29 Thread Greg Steuck
Another one bit the dust, ci-openbsd-main-0 this time. ci-openbsd$ vmctl status ID PID VCPUS MAXMEM CURMEM TTYOWNER NAME 390 35277 12.0G474M ttyp2syzkaller ci-openbsd-main-2 1 - 1512M - -syzkaller syzkaller ci-openbsd$ ps axu

vmd losing VMs

2018-09-28 Thread Greg Steuck
I've been running syzkaller for about a day now. It launches/kills VMs all the time. Somewhere along the way vmd seems to have lost track of one of its VMs. Notice how syzkaller ci-openbsd-main-1 is conspicuously missing from vmctl status even though there's a process chewing the CPU for it. Some