Re: vmd: VMs are stopping with: vcpu 0 run ioctl failed: Invalid argument
On Wed, Jan 17, 2018 at 01:45:15PM +, Olivier Cherrier wrote: > On Wed, Jan 17, 2018 at 09:30:18AM +, o...@symacx.com wrote: > > > This looks like a custom kernel, can you run the latest snapshot, as-is? > > > > The only change I made is related to /sys/dev/ic/pckbc.c in order to > > have a working trackpad. > > > > > Might want to either comment out the interface for "uplink" or remove it > > > altogether. > > > > OK, commented out. > > I am upgrading the physical machine with VMM_DEBUG. > > > Now, everything is running current. > After reboot, vmd started the am-azure VM. It worked fine for a few > hours and it just stopped. Here are the kernel messages: > > Jan 17 14:35:29 stanislas /bsd: vmx_handle_exit: unhandled exit 0x31 (EPT > misconfiguration) > Jan 17 14:35:29 stanislas vmd[86737]: vcpu_run_loop: vm 1 / vcpu 0 run ioctl > failed: Invalid argument > Jan 17 14:35:30 stanislas /bsd: vmm_free_vpid: freed VPID/ASID 1 > > I hope it helps. > Thanks > > -- > Olivier Cherrier > Phone: +352691754777 > mailto:o...@symacx.com > There seems to be a regression in the last few weeks or so. mpi@ is seeing this, Olivier is seeing this, and now I'm seeing it too. I'll see if I can track it down. -ml
Re: vmd: VMs are stopping with: vcpu 0 run ioctl failed: Invalid argument
On Wed, Jan 17, 2018 at 09:30:18AM +, o...@symacx.com wrote: > > This looks like a custom kernel, can you run the latest snapshot, as-is? > > The only change I made is related to /sys/dev/ic/pckbc.c in order to > have a working trackpad. > > > Might want to either comment out the interface for "uplink" or remove it > > altogether. > > OK, commented out. > I am upgrading the physical machine with VMM_DEBUG. Now, everything is running current. After reboot, vmd started the am-azure VM. It worked fine for a few hours and it just stopped. Here are the kernel messages: Jan 17 14:35:29 stanislas /bsd: vmx_handle_exit: unhandled exit 0x31 (EPT misconfiguration) Jan 17 14:35:29 stanislas vmd[86737]: vcpu_run_loop: vm 1 / vcpu 0 run ioctl failed: Invalid argument Jan 17 14:35:30 stanislas /bsd: vmm_free_vpid: freed VPID/ASID 1 I hope it helps. Thanks -- Olivier Cherrier Phone: +352691754777 mailto:o...@symacx.com
Re: vmd: VMs are stopping with: vcpu 0 run ioctl failed: Invalid argument
On Tue, Jan 16, 2018 at 09:20:57PM -0800, cardena...@gmail.com wrote:
> > Only OpenBSD involved here. The physical machine's dmesg is attached.
> > My /etc/vm.conf configuration file is also attached.
> >
> > I don't think it is related to memory exhaustion since it sometimes
> > happens when the machine is not loaded (cpu and memory).
> >
> > > I would add:
> > > * What was your "vmctl start ..." command?
> >
> > Here, I did use : vmd -d
> >
> > Will try by recompiling with VMM_DEBUG.
> >
> > Thanks a lot,
> > Best
> > Olivier
>
> Ok...see comments inline
>
> > OpenBSD 6.2-current (GENERIC.MP) #2: Thu Dec 7 10:13:35 CET 2017
> > o...@stanislas.symacx.com:/sys/arch/amd64/compile/GENERIC.MP
> This looks like a custom kernel, can you run the latest snapshot, as-is?
The only change I made is related to /sys/dev/ic/pckbc.c in order to
have a working trackpad.
> > vm "openbsd.vm" {
> > disable
> > boot $sets "bsd.rd"
> >
> > # First disk from 'vmctl create "/home/vm/OpenBSD.img" -s 4G'
> > disk "/home/vm/OpenBSD.img"
> >
> > # Second disk from OpenBSD contains the install sets
> > disk $sets "install60.fs"
> >
> > # Interface will show up as tap(4) on the host and as vio(4) in the VM
> > interface { switch "uplink" }
> > interface { switch "local" }
> > }
> Might want to either comment out the interface for "uplink" or remove it
> altogether.
OK, commented out.
> > vm "puffy1" {
> > owner oc
> > disable
> > memory 512M
> > boot $sets "bsd"
> > disk $sets "puffy1.img"
> >
> > # Use a specific tap(4) interface with a hardcoded MAC address
> > # local interface tap
> > interface {
> > switch "local"
> > lladdr 00:11:22:33:44:55
> > }
> > }
> > vm "puffy2" {
> > owner oc
> > disable
> > memory 1024M
> > # boot $sets "current/bsd.rd"
> > disk $sets "puffy2.img"
> > disk $sets "puffy2_data0.img"
> >
> > # Use a specific tap(4) interface with a hardcoded MAC address
> > # local interface tap
> > interface {
> > switch "local"
> > lladdr 00:11:22:33:44:56
> > }
> > }
> > vm "am-azure" {
> > owner oc
> > memory 1024M
> > # boot $sets "current/bsd.rd"
> > disk $sets "am-azure.img"
> > disk $sets "am-azure_data0.img"
> >
> > # Use a specific tap(4) interface with a hardcoded MAC address
> > # local interface tap
> > interface {
> > switch "local"
> > lladdr 00:11:22:33:44:57
> > }
> > }
>
> Ok...it's unclear which version the guest(s) are running. Can you
> have the guest run the latest snapshot (same as the host) so we can
> narrow the variables?
Guests are running various versions of current. I have this problem with
the latest snapshot as well:
$ sysctl kern.version
kern.version=OpenBSD 6.2-current (GENERIC) #359: Mon Jan 15 22:32:02 MST 2018
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
It happens if I run only puffy1, puffy1 and puffy2 or only am-azure.
> Also, which vm was giving issue?
All of them.
I am upgrading the physical machine with VMM_DEBUG.
BTW, congratulations for the hardware support (ACPI, ...). This laptop
goes to suspend multiple times a day and everything still works after
more than a month:
10:23:43 oc@stanislas $ uptime
10:23AM up 41 days, 9 mins, 16 users, load averages: 1.01, 1.23, 0.97
10:23:44 oc@stanislas $
Thanks
--
Olivier Cherrier
Phone: +352691754777
mailto:o...@symacx.com
Re: vmd: VMs are stopping with: vcpu 0 run ioctl failed: Invalid argument
On Tue, Jan 16, 2018 at 06:24:18PM +, Olivier Cherrier wrote: > On Tue, Jan 16, 2018 at 06:56:04AM -0800, cardena...@gmail.com wrote: > > > Here is what I typically get : > > > > > > ... > > > vionet queue notify - no space, dropping packet > > > vionet queue notify - no space, dropping packet > > > vionet queue notify - no space, dropping packet > > > vionet queue notify - no space, dropping packet > > > vionet queue notify - no space, dropping packet > > > vionet queue notify - no space, dropping packet > > > vionet queue notify - no space, dropping packet > > > vcpu_run_loop: vm 23 / vcpu 0 run ioctl failed: Invalid argument > > > vmm_sighdlr: handling signal 20 > > > vmm_sighdlr: attempting to terminate vm 4 > > > terminate_vm: terminating vmid 23 > > > vmm_sighdlr: calling vm_remove > > > vm_remove: removing vm id 4 from running config > > > vm_remove: calling vm_stop > > > vm_stop: stopping vm 4 > > > vmd_dispatch_vmm: handling TERMINATE_EVENT for vm id 4 ret 22 > > > vmd_dispatch_vmm: about to stop vm id 4 > > > vm_stop: stopping vm 4 > > > > > > > to enable VMM_DEBUG in sys/arch/amd64/amd64/vmm.c and rebuild the > > > > kernel. > > > > > > Will try that. > > > > > > Thank you! > > > Best > > > Olivier > > > > > > > Thanks for reporting the symptoms Olivier. > > > > In order for us to reproduce this and find the root cause, we're going > > to need the info that Mike was asking: > > * What guest VM are you running here? > > * How much memory are you assigning? > > Hi Carlos, > > Only OpenBSD involved here. The physical machine's dmesg is attached. > My /etc/vm.conf configuration file is also attached. > > I don't think it is related to memory exhaustion since it sometimes > happens when the machine is not loaded (cpu and memory). > > > I would add: > > * What was your "vmctl start ..." command? > > Here, I did use : vmd -d > > Will try by recompiling with VMM_DEBUG. > > Thanks a lot, > Best > Olivier Ok...see comments inline > OpenBSD 6.2-current (GENERIC.MP) #2: Thu Dec 7 10:13:35 CET 2017 > o...@stanislas.symacx.com:/sys/arch/amd64/compile/GENERIC.MP This looks like a custom kernel, can you run the latest snapshot, as-is? > real mem = 4188946432 (3994MB) > avail mem = 4054945792 (3867MB) > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x87ee6000 (43 entries) > bios0: vendor American Megatrends Inc. version "5.11" date 06/03/2016 > bios0: Notebook N24_25JU > acpi0 at bios0: rev 2 > acpi0: sleep states S0 S3 S4 S5 > acpi0: tables DSDT FACP APIC FPDT FIDT MCFG HPET SSDT SSDT DBGP DBG2 SSDT > UEFI SSDT DMAR ASF! > acpi0: wakeup devices PEGP(S4) PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) > PXSX(S4) RP17(S4) PXSX(S4) RP18(S4) PXSX(S4) RP19(S4) PXSX(S4) RP20(S4) > PXSX(S4) RP01(S4) [...] > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.80 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges > cpu0: apic clock running at 24MHz > cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE > cpu1 at mainbus0: apid 2 (application processor) > cpu1: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT > cpu1: 256KB 64b/line 8-way L2 cache > cpu1: smt 0, core 1, package 0 > cpu2 at mainbus0: apid 1 (application processor) > cpu2: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz > cpu2: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT > cpu2: 256KB 64b/line 8-way L2 cache > cpu2: smt 1, core 0, package 0 > cpu3 at mainbus0: apid 3 (application processor) > cpu3: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz > cpu3: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,S
Re: vmd: VMs are stopping with: vcpu 0 run ioctl failed: Invalid argument
On Tue, Jan 16, 2018 at 06:56:04AM -0800, cardena...@gmail.com wrote: > > Here is what I typically get : > > > > ... > > vionet queue notify - no space, dropping packet > > vionet queue notify - no space, dropping packet > > vionet queue notify - no space, dropping packet > > vionet queue notify - no space, dropping packet > > vionet queue notify - no space, dropping packet > > vionet queue notify - no space, dropping packet > > vionet queue notify - no space, dropping packet > > vcpu_run_loop: vm 23 / vcpu 0 run ioctl failed: Invalid argument > > vmm_sighdlr: handling signal 20 > > vmm_sighdlr: attempting to terminate vm 4 > > terminate_vm: terminating vmid 23 > > vmm_sighdlr: calling vm_remove > > vm_remove: removing vm id 4 from running config > > vm_remove: calling vm_stop > > vm_stop: stopping vm 4 > > vmd_dispatch_vmm: handling TERMINATE_EVENT for vm id 4 ret 22 > > vmd_dispatch_vmm: about to stop vm id 4 > > vm_stop: stopping vm 4 > > > > > to enable VMM_DEBUG in sys/arch/amd64/amd64/vmm.c and rebuild the kernel. > > > > Will try that. > > > > Thank you! > > Best > > Olivier > > > > Thanks for reporting the symptoms Olivier. > > In order for us to reproduce this and find the root cause, we're going > to need the info that Mike was asking: > * What guest VM are you running here? > * How much memory are you assigning? Hi Carlos, Only OpenBSD involved here. The physical machine's dmesg is attached. My /etc/vm.conf configuration file is also attached. I don't think it is related to memory exhaustion since it sometimes happens when the machine is not loaded (cpu and memory). > I would add: > * What was your "vmctl start ..." command? Here, I did use : vmd -d Will try by recompiling with VMM_DEBUG. Thanks a lot, Best Olivier OpenBSD 6.2-current (GENERIC.MP) #2: Thu Dec 7 10:13:35 CET 2017 o...@stanislas.symacx.com:/sys/arch/amd64/compile/GENERIC.MP real mem = 4188946432 (3994MB) avail mem = 4054945792 (3867MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x87ee6000 (43 entries) bios0: vendor American Megatrends Inc. version "5.11" date 06/03/2016 bios0: Notebook N24_25JU acpi0 at bios0: rev 2 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT MCFG HPET SSDT SSDT DBGP DBG2 SSDT UEFI SSDT DMAR ASF! acpi0: wakeup devices PEGP(S4) PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PXSX(S4) RP17(S4) PXSX(S4) RP18(S4) PXSX(S4) RP19(S4) PXSX(S4) RP20(S4) PXSX(S4) RP01(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.80 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 24MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 1 (application processor) cpu2: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 1, core 0, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 1, core 1, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec0,
Re: vmd: VMs are stopping with: vcpu 0 run ioctl failed: Invalid argument
On Tue, Jan 16, 2018 at 09:40:54AM +, Olivier Cherrier wrote: > On Thu, Dec 21, 2017 at 10:57:12AM -0800, mlar...@azathoth.net wrote: > > > Hi, > > > > > > I am running ~ current on amd64 (dmesg attached) and am experiencing > > > vmd crashes. The processes responsible for operating the VMs are > > > disappearing and I get the following messages in deamon: > > > > > > Dec 20 12:26:32 stanislas vmd[38693]: vcpu_run_loop: vm 14 / vcpu 0 run > > > ioctl failed: Invalid argument > > > Dec 20 12:26:32 stanislas vmd[98389]: vcpu_run_loop: vm 15 / vcpu 0 run > > > ioctl failed: Invalid argument > > > > > > It happens randomly. Generally the VMs are staying up for a few hours. > > > Is there anything I can do to troubleshoot ? > > > > > > > Could be anything. What guest VM are you running here? And how much memory > > are > > you assigning? > > > > A useful option is to run vmd manually (rcctl stop vmd && vmd -d), > > and/or > > Here is what I typically get : > > ... > vionet queue notify - no space, dropping packet > vionet queue notify - no space, dropping packet > vionet queue notify - no space, dropping packet > vionet queue notify - no space, dropping packet > vionet queue notify - no space, dropping packet > vionet queue notify - no space, dropping packet > vionet queue notify - no space, dropping packet > vcpu_run_loop: vm 23 / vcpu 0 run ioctl failed: Invalid argument > vmm_sighdlr: handling signal 20 > vmm_sighdlr: attempting to terminate vm 4 > terminate_vm: terminating vmid 23 > vmm_sighdlr: calling vm_remove > vm_remove: removing vm id 4 from running config > vm_remove: calling vm_stop > vm_stop: stopping vm 4 > vmd_dispatch_vmm: handling TERMINATE_EVENT for vm id 4 ret 22 > vmd_dispatch_vmm: about to stop vm id 4 > vm_stop: stopping vm 4 > > > to enable VMM_DEBUG in sys/arch/amd64/amd64/vmm.c and rebuild the kernel. > > Will try that. > > Thank you! > Best > Olivier > Thanks for reporting the symptoms Olivier. In order for us to reproduce this and find the root cause, we're going to need the info that Mike was asking: * What guest VM are you running here? * How much memory are you assigning? I would add: * What was your "vmctl start ..." command? Thanks in advance. +--+ Carlos
Re: vmd: VMs are stopping with: vcpu 0 run ioctl failed: Invalid argument
On Thu, Dec 21, 2017 at 10:57:12AM -0800, mlar...@azathoth.net wrote: > > Hi, > > > > I am running ~ current on amd64 (dmesg attached) and am experiencing > > vmd crashes. The processes responsible for operating the VMs are > > disappearing and I get the following messages in deamon: > > > > Dec 20 12:26:32 stanislas vmd[38693]: vcpu_run_loop: vm 14 / vcpu 0 run > > ioctl failed: Invalid argument > > Dec 20 12:26:32 stanislas vmd[98389]: vcpu_run_loop: vm 15 / vcpu 0 run > > ioctl failed: Invalid argument > > > > It happens randomly. Generally the VMs are staying up for a few hours. > > Is there anything I can do to troubleshoot ? > > > > Could be anything. What guest VM are you running here? And how much memory are > you assigning? > > A useful option is to run vmd manually (rcctl stop vmd && vmd -d), and/or Here is what I typically get : ... vionet queue notify - no space, dropping packet vionet queue notify - no space, dropping packet vionet queue notify - no space, dropping packet vionet queue notify - no space, dropping packet vionet queue notify - no space, dropping packet vionet queue notify - no space, dropping packet vionet queue notify - no space, dropping packet vcpu_run_loop: vm 23 / vcpu 0 run ioctl failed: Invalid argument vmm_sighdlr: handling signal 20 vmm_sighdlr: attempting to terminate vm 4 terminate_vm: terminating vmid 23 vmm_sighdlr: calling vm_remove vm_remove: removing vm id 4 from running config vm_remove: calling vm_stop vm_stop: stopping vm 4 vmd_dispatch_vmm: handling TERMINATE_EVENT for vm id 4 ret 22 vmd_dispatch_vmm: about to stop vm id 4 vm_stop: stopping vm 4 > to enable VMM_DEBUG in sys/arch/amd64/amd64/vmm.c and rebuild the kernel. Will try that. Thank you! Best Olivier
Re: vmd: VMs are stopping with: vcpu 0 run ioctl failed: Invalid argument
On Thu, Dec 21, 2017 at 08:36:19AM +, Olivier Cherrier wrote: > > Hi, > > I am running ~ current on amd64 (dmesg attached) and am experiencing > vmd crashes. The processes responsible for operating the VMs are > disappearing and I get the following messages in deamon: > > Dec 20 12:26:32 stanislas vmd[38693]: vcpu_run_loop: vm 14 / vcpu 0 run ioctl > failed: Invalid argument > Dec 20 12:26:32 stanislas vmd[98389]: vcpu_run_loop: vm 15 / vcpu 0 run ioctl > failed: Invalid argument > > It happens randomly. Generally the VMs are staying up for a few hours. > Is there anything I can do to troubleshoot ? > Could be anything. What guest VM are you running here? And how much memory are you assigning? A useful option is to run vmd manually (rcctl stop vmd && vmd -d), and/or to enable VMM_DEBUG in sys/arch/amd64/amd64/vmm.c and rebuild the kernel. Obviously that kernel goes on the host. -ml > Thank you, > Best > > -- > Olivier Cherrier > Phone: +352691754777 > mailto:o...@symacx.com > OpenBSD 6.2-current (GENERIC.MP) #2: Thu Dec 7 10:13:35 CET 2017 > o...@stanislas.symacx.com:/sys/arch/amd64/compile/GENERIC.MP > real mem = 4188946432 (3994MB) > avail mem = 4054945792 (3867MB) > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x87ee6000 (43 entries) > bios0: vendor American Megatrends Inc. version "5.11" date 06/03/2016 > bios0: Notebook N24_25JU > acpi0 at bios0: rev 2 > acpi0: sleep states S0 S3 S4 S5 > acpi0: tables DSDT FACP APIC FPDT FIDT MCFG HPET SSDT SSDT DBGP DBG2 SSDT > UEFI SSDT DMAR ASF! > acpi0: wakeup devices PEGP(S4) PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) > PXSX(S4) RP17(S4) PXSX(S4) RP18(S4) PXSX(S4) RP19(S4) PXSX(S4) RP20(S4) > PXSX(S4) RP01(S4) [...] > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.80 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges > cpu0: apic clock running at 24MHz > cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE > cpu1 at mainbus0: apid 2 (application processor) > cpu1: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT > cpu1: 256KB 64b/line 8-way L2 cache > cpu1: smt 0, core 1, package 0 > cpu2 at mainbus0: apid 1 (application processor) > cpu2: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz > cpu2: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT > cpu2: 256KB 64b/line 8-way L2 cache > cpu2: smt 1, core 0, package 0 > cpu3 at mainbus0: apid 3 (application processor) > cpu3: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz > cpu3: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT > cpu3: 256KB 64b/line 8-way L2 cache > cpu3: smt 1, core 1, package 0 > ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 120 pins > acpimcfg0 at acpi0 addr 0xe000, bus 0-255 > acpihpet0 at acpi0: 2399 Hz > acpiprt0 at acpi0: bus 0 (PCI0) > acpiprt1 at acpi0: bus -1 (PEG0) > acpiprt2 at acpi0: bus -1 (PEG1) > acpiprt3 at acpi0: bus -1 (PEG2) > acpiprt4 at acpi0: bus -1 (RP17) > acpiprt5 at acpi0: bus -1 (RP18) > acpiprt6 at acpi0: bus -1 (RP19) > acpiprt7 at acpi0: bus -1 (RP20) > acpiprt8 at acpi0: bus 1 (RP01) > acpiprt9 at acpi0: bus 2 (RP02) > acpiprt10 at acpi0: bus -1 (RP03) > acpiprt11 at acpi0: bus -1 (RP04) > acpiprt12 at acpi0: bus -1 (RP05) > acpiprt13 at acpi0: bus -1 (RP06) > acpiprt14 at acpi0: bus -1 (RP07) > acpiprt15 at acpi0: bus -1 (RP08) >
vmd: VMs are stopping with: vcpu 0 run ioctl failed: Invalid argument
Hi, I am running ~ current on amd64 (dmesg attached) and am experiencing vmd crashes. The processes responsible for operating the VMs are disappearing and I get the following messages in deamon: Dec 20 12:26:32 stanislas vmd[38693]: vcpu_run_loop: vm 14 / vcpu 0 run ioctl failed: Invalid argument Dec 20 12:26:32 stanislas vmd[98389]: vcpu_run_loop: vm 15 / vcpu 0 run ioctl failed: Invalid argument It happens randomly. Generally the VMs are staying up for a few hours. Is there anything I can do to troubleshoot ? Thank you, Best -- Olivier Cherrier Phone: +352691754777 mailto:o...@symacx.com OpenBSD 6.2-current (GENERIC.MP) #2: Thu Dec 7 10:13:35 CET 2017 o...@stanislas.symacx.com:/sys/arch/amd64/compile/GENERIC.MP real mem = 4188946432 (3994MB) avail mem = 4054945792 (3867MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x87ee6000 (43 entries) bios0: vendor American Megatrends Inc. version "5.11" date 06/03/2016 bios0: Notebook N24_25JU acpi0 at bios0: rev 2 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT MCFG HPET SSDT SSDT DBGP DBG2 SSDT UEFI SSDT DMAR ASF! acpi0: wakeup devices PEGP(S4) PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PXSX(S4) RP17(S4) PXSX(S4) RP18(S4) PXSX(S4) RP19(S4) PXSX(S4) RP20(S4) PXSX(S4) RP01(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.80 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 24MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 1 (application processor) cpu2: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 1, core 0, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Pentium(R) CPU 4405U @ 2.10GHz, 2095.14 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 1, core 1, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 120 pins acpimcfg0 at acpi0 addr 0xe000, bus 0-255 acpihpet0 at acpi0: 2399 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus -1 (PEG0) acpiprt2 at acpi0: bus -1 (PEG1) acpiprt3 at acpi0: bus -1 (PEG2) acpiprt4 at acpi0: bus -1 (RP17) acpiprt5 at acpi0: bus -1 (RP18) acpiprt6 at acpi0: bus -1 (RP19) acpiprt7 at acpi0: bus -1 (RP20) acpiprt8 at acpi0: bus 1 (RP01) acpiprt9 at acpi0: bus 2 (RP02) acpiprt10 at acpi0: bus -1 (RP03) acpiprt11 at acpi0: bus -1 (RP04) acpiprt12 at acpi0: bus -1 (RP05) acpiprt13 at acpi0: bus -1 (RP06) acpiprt14 at acpi0: bus -1 (RP07) acpiprt15 at acpi0: bus -1 (RP08) acpiprt16 at acpi0: bus 3 (RP09) acpiprt17 at acpi0: bus -1 (RP10) acpiprt18 at acpi0: bus -1 (RP11) acpiprt19 at acpi0: bus -1 (RP12) acpiprt20 at acpi0: bus -1 (RP13) acpiprt21 at acpi0: bus -1 (RP14) acpiprt22 at acpi0: bus -1 (RP15) acpiprt23 at acpi0: bus -1 (RP16) acpiec0 at acpi0 acpicpu0 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS acpicpu1 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS acpicpu2 at acpi0: C3(200@1034 mwait.1@0x60), C2(200
