The below diff fixes an issue reported by kn@ on bugs@ [1]. joshe@ also
observed the issue and confirmed the below diff resolves it.
The symptoms were quite odd: errors from fdc(4) during an OpenBSD guest
booting under vmm(4)/vmd(8). We don't emulate a floppy disk drive!!!
I introduced a bug in r1.287 [2] when simplifying parts of
vcpu_run_{svm,vmx} by letting the functions return 0 instead of
voluntarily yielding. The edge case I didn't account for is if after a
vmexit for an IN instruction, the io port address isn't one emulated by
vmd(8) in userland, vmm(4) will perform the emulation (not the bug) by
writing the appropriate number of 0xff bytes to AL/AX/EAX. IF the
scheduler would like us to yield, we return setting a vrp exit code of
VM_EXIT_NONE (since we aren't asking userland/vmd to help with any
emulation).
vmd(8) correctly handles this exit, but vmm(4) never copies out the
current vcpu registers and irqready state. When vmd(8) runs the vcpu
again, the vcpu's guest state still has a vmexit related to the IO
operation and presumes vmd(8) modified RAX and overwrites the vcpu's
RAX before re-entering the guest.
This behavior occurs on both Intel and AMD. To confirm, I added some
printfs to fdc(4) and specifically checked when the dma reads returned
something other than 0xff on instances of both types of host. (Since
it's probabilistic, it's not uncommon to see it happen only 3-4 times
out of the 100k bus reads out_fdc() attempts, but it seems more
reproducible on older hardware.)
ok?
-dv
[1] https://marc.info/?l=openbsd-bugs&m=163682062027764&w=2
[2]
https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/arch/amd64/amd64/vmm.c.diff?r1=1.286&r2=1.287
Index: sys/arch/amd64/amd64/vmm.c
===================================================================
RCS file: /opt/cvs/src/sys/arch/amd64/amd64/vmm.c,v
retrieving revision 1.294
diff -u -p -r1.294 vmm.c
--- sys/arch/amd64/amd64/vmm.c 26 Oct 2021 16:29:49 -0000 1.294
+++ sys/arch/amd64/amd64/vmm.c 20 Nov 2021 21:46:07 -0000
@@ -4301,9 +4301,10 @@ vm_run(struct vm_run_params *vrp)
rw_exit_write(&vmm_softc->vm_lock);
}
ret = 0;
- } else if (ret == EAGAIN) {
+ } else if (ret == 0 || ret == EAGAIN) {
/* If we are exiting, populate exit data so vmd can help. */
- vrp->vrp_exit_reason = vcpu->vc_gueststate.vg_exit_reason;
+ vrp->vrp_exit_reason = (ret == 0) ? VM_EXIT_NONE
+ : vcpu->vc_gueststate.vg_exit_reason;
vrp->vrp_irqready = vcpu->vc_irqready;
vcpu->vc_state = VCPU_STATE_STOPPED;
@@ -4312,9 +4313,6 @@ vm_run(struct vm_run_params *vrp)
ret = EFAULT;
} else
ret = 0;
- } else if (ret == 0) {
- vrp->vrp_exit_reason = VM_EXIT_NONE;
- vcpu->vc_state = VCPU_STATE_STOPPED;
} else {
vrp->vrp_exit_reason = VM_EXIT_TERMINATED;
vcpu->vc_state = VCPU_STATE_TERMINATED;