On Thu, Aug 02, 2018 at 09:22:53AM +0200, Nulani t'Acraya wrote: > Hello, > > Something similar also appears to also be affecting bhyve, at least on an > AMD Opteron 4228 HE. The error produced is different depending on > whether bhyve is instructed to ignore accessed to model specific registers > that are not implemented in the current CPU. I haven't had to have that flag > toggled previously. I've included the dmesg and trace from both setups below. > > A snapshot of -current with a build date of 1533181438 - Thu Aug 2 03:43:58 > UTC 2018 boots successfully with ignore_bad_msr set to on. I'm not entirely > sure if Bryan's patch will have made it into that snapshot or not, but > if it has, > it appears to also be fixing the issue on bhyve. Thanks! > > Sincerely, > Nulani. >
The problem is not ignoring access to bad MSRs. The problem is that these hypervisors dont know that the MSR we are accessing is indeed a real, valid MSR. And then they panic. -ml > ### 6.3 without ignore_bad_msr ### > Copyright (c) 1982, 1986, 1989, 1991, 1993 > The Regents of the University of California. All rights reserved. > Copyright (c) 1995-2018 OpenBSD. All rights reserved. https://www.OpenBSD.org > > OpenBSD 6.3 (GENERIC) #7: Sun Jul 29 11:30:47 CEST 2018 > r...@syspatch-63-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC > real mem = 1056964608 (1008MB) > avail mem = 1019158528 (971MB) > warning: no entropy supplied by boot loader > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xf101f (9 entries) > bios0: vendor BHYVE version "1.00" date 03/14/2014 > bios0: bhyve BHYVE > acpi0 at bios0: rev 2 > acpi0: sleep states S5 > acpi0: tables DSDT APIC FACP HPET MCFG > acpi0: wakeup devices > acpitimer0 at acpi0: 3579545 Hz, 32 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: AMD Opteron(tm) Processor 4228 HE, 2800.42 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,MMXX,FFXSR,PAGE1GB,LONG,LAHF,CMPLEG,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,XOP,SKINIT,WDT,FMA4,ITSC > cpu0: 64KB 64b/line 2-way I-cache, 16KB 64b/line 4-way D-cache, 2MB > 64b/line 16-way L2 cache, 8MB 64b/line 64-way L3 cache > cpu0: ITLB 48 4KB entries fully associative, 24 4MB entries fully associative > cpu0: DTLB 32 4KB entries fully associative, 32 4MB entries fully associative > kernel: protection fault trap, code=0 > Stopped at 0xffffffff81219c59: wrmsr > ddb> trace > ffffffff81219c59(ffff800000031700,ffffffff81a7fff0,ffffffff81a7d028,ffffffff81c > 06a58,ffff800000031724,0) at 0xffffffff81219c59 > ffffffff81008d2e(ffff800000023100,ffffffff81c06a58,ffff800000031700,ffffffff81a > 7d000,ffffffff81008d2e,ffffffff81c069b0) at 0xffffffff81008d2e > ffffffff813618b8(0,ffff8000000232c4,ffff800000023298,ffff8000000232c4,ffffffff8 > 1c06a38,ffffffff816c51a0) at 0xffffffff813618b8 > ffffffff816c4c36(ffff800000020400,ffffffff81c06b60,ffffffff81ab3f38,ffff8000000 > 31200,ffff800000031224,0) at 0xffffffff816c4c36 > ffffffff813618b8(ffffffff81c06b60,ffff800000020400,ffff800000020470,ffff8000000 > 20460,ffff800000023280,ffffffff8100d040) at 0xffffffff813618b8 > ffffffff8100c571(ffff800000023180,ffffffff81c06c50,ffffffff81a811a8,ffff8000000 > 20400,ffff800000020424,0) at 0xffffffff8100c571 > ffffffff813618b8(ffff800014a67023,ffff800000023180,3c,104,ffff800014a67042,ffff > ffff8140b6f0) at 0xffffffff813618b8 > ffffffff8140a766(ffff800000023100,ffffffff81c06d88,ffffffff81aa1ea8,ffff8000000 > 23180,ffff8000000231a4,0) at 0xffffffff8140a766 > ffffffff813618b8(ffffffff81c06d88,ffff800000023100,ffffffff81a8ba98,ffff8000000 > 23100,ffff800000023124,ffffffff811a9830) at 0xffffffff813618b8 > ffffffff811a95a1(0,0,0,ffffffff81c06db0,ffffffff81c06e20,3000000010) at > 0xfffff > fff811a95a1 > ffffffff813618b8(0,ffffffff81827131,ffffffff81a9fd8a,ffffffff81c06e78,b28,0) > at > 0xffffffff813618b8 > ffffffff81361a53(0,0,0,0,ffffffff81c00008,0) at 0xffffffff81361a53 > ffffffff8101187b(0,0,ffffffff8101187b,ffffffff81c06ef0,0,0) at > 0xffffffff810118 > 7b > ffffffff8116b8c3(0,0,0,0,ffffffff8116b8c3,ffffffff81c06f20) at > 0xffffffff8116b8 > c3 > end trace frame: 0x0, count: -14 > ddb> ps > PID TID PPID UID S FLAGS WAIT COMMAND > * 0 0 -1 0 7 0x10200 swapper > > ###6.3 with ignore_bad_msr### > Copyright (c) 1982, 1986, 1989, 1991, 1993 > The Regents of the University of California. All rights reserved. > Copyright (c) 1995-2018 OpenBSD. All rights reserved. https://www.OpenBSD.org > > OpenBSD 6.3 (GENERIC) #7: Sun Jul 29 11:30:47 CEST 2018 > r...@syspatch-63-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC > real mem = 1056964608 (1008MB) > avail mem = 1019158528 (971MB) > warning: no entropy supplied by boot loader > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xf101f (9 entries) > bios0: vendor BHYVE version "1.00" date 03/14/2014 > bios0: bhyve BHYVE > acpi0 at bios0: rev 2 > acpi0: sleep states S5 > acpi0: tables DSDT APIC FACP HPET MCFG > acpi0: wakeup devices > acpitimer0 at acpi0: 3579545 Hz, 32 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: AMD Opteron(tm) Processor 4228 HE, 2800.54 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,MMXX,FFXSR,PAGE1GB,LONG,LAHF,CMPLEG,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,XOP,SKINIT,WDT,FMA4,ITSC > cpu0: 64KB 64b/line 2-way I-cache, 16KB 64b/line 4-way D-cache, 2MB > 64b/line 16-way L2 cache, 8MB 64b/line 64-way L3 cache > cpu0: ITLB 48 4KB entries fully associative, 24 4MB entries fully associative > cpu0: DTLB 32 4KB entries fully associative, 32 4MB entries fully associative > kernel: protection fault trap, code=0 > Stopped at 0xffffffff81219c59: wrmsr > ddb> trace > ffffffff81219c59(ffff800000031700,ffffffff81a7fff0,ffffffff81a7d028,ffffffff81c > 06a58,ffff800000031724,0) at 0xffffffff81219c59 > ffffffff81008d2e(ffff800000023100,ffffffff81c06a58,ffff800000031700,ffffffff81a > 7d000,ffffffff81008d2e,ffffffff81c069b0) at 0xffffffff81008d2e > ffffffff813618b8(0,ffff8000000232c4,ffff800000023298,ffff8000000232c4,ffffffff8 > 1c06a38,ffffffff816c51a0) at 0xffffffff813618b8 > ffffffff816c4c36(ffff800000020400,ffffffff81c06b60,ffffffff81ab3f38,ffff8000000 > 31200,ffff800000031224,0) at 0xffffffff816c4c36 > ffffffff813618b8(ffffffff81c06b60,ffff800000020400,ffff800000020470,ffff8000000 > 20460,ffff800000023280,ffffffff8100d040) at 0xffffffff813618b8 > ffffffff8100c571(ffff800000023180,ffffffff81c06c50,ffffffff81a811a8,ffff8000000 > 20400,ffff800000020424,0) at 0xffffffff8100c571 > ffffffff813618b8(ffff800014a67023,ffff800000023180,3c,104,ffff800014a67042,ffff > ffff8140b6f0) at 0xffffffff813618b8 > ffffffff8140a766(ffff800000023100,ffffffff81c06d88,ffffffff81aa1ea8,ffff8000000 > 23180,ffff8000000231a4,0) at 0xffffffff8140a766 > ffffffff813618b8(ffffffff81c06d88,ffff800000023100,ffffffff81a8ba98,ffff8000000 > 23100,ffff800000023124,ffffffff811a9830) at 0xffffffff813618b8 > ffffffff811a95a1(0,0,0,ffffffff81c06db0,ffffffff81c06e20,3000000010) at > 0xfffff > fff811a95a1 > ffffffff813618b8(0,ffffffff81827131,ffffffff81a9fd8a,ffffffff81c06e78,b28,0) > at > 0xffffffff813618b8 > ffffffff81361a53(0,0,0,0,ffffffff81c00008,0) at 0xffffffff81361a53 > ffffffff8101187b(0,0,ffffffff8101187b,ffffffff81c06ef0,0,0) at > 0xffffffff810118 > 7b > ffffffff8116b8c3(0,0,0,0,ffffffff8116b8c3,ffffffff81c06f20) at > 0xffffffff8116b8 > c3 > end trace frame: 0x0, count: -14 > ddb> ps > PID TID PPID UID S FLAGS WAIT COMMAND > * 0 0 -1 0 7 0x10200 swapper > > On 1 August 2018 at 23:11, Bryan Steele <bry...@gmail.com> wrote: > > On Wed, Aug 01, 2018 at 01:07:33PM -0700, Mike Larkin wrote: > >> On Wed, Aug 01, 2018 at 12:14:59PM -0400, Bryan Steele wrote: > >> > On Wed, Aug 01, 2018 at 11:27:26AM -0400, Bryan Steele wrote: > >> > > On Wed, Aug 01, 2018 at 03:46:25PM +0200, Elmer Skjødt Henriksen wrote: > >> > > > After installing the 014_amdlfence patch released yesterday for 6.3, > >> > > > my > >> > > > OpenBSD VM crashes on boot. It's running under KVM on a Linux box > >> > > > (Ubuntu > >> > > > 18.04 w/ kernel 4.15) on an AMD Ryzen 7 1700 (microcode 0x8001137). > >> > > > I suppose this would also happen on vmm(4) and bhyve, however I > >> > > > don't have > >> > > > any such AMD hosts available for testing. > >> > > > >> > > Hi Elmer, > >> > > > >> > > This was tested in vmm(4), which does work, unfortunately there was not > >> > > extensive testing by in other virtualization software. The MSR that is > >> > > being set here is only mentioned in AMDs whitepaper and I had no reason > >> > > to believe any special consideration was needed for guest VMs on AMD > >> > > processors. > >> > > > >> > > > It occurs both using libvirt's "EPYC" CPU model and using > >> > > > "host-passthrough" > >> > > > (i.e. no virtual CPU model), but the "core2duo" CPU model works fine. > >> > > > > >> > > > I guess not many people are running OpenBSD as a VM, and even less > >> > > > on AMD > >> > > > hardware. But still, a syspatch leaving the system unable to boot is > >> > > > probably not a good thing. :) > >> > > > > >> > > > >> > > Even so, I would like to apologize. This situation is unfortunate, and > >> > > I'll try to work with other developers to find the best way forward. > >> > > But, I regret I am only but an amateur magician. > >> > > > >> > > -Bryan. > >> > > >> > Actually, it looks like this is at least partially a KVM/QEMU bug. In > >> > the meantime I guess the solution would be to do as you suggested and > >> > set a different CPU model for now until Linux distros include a fix for > >> > this. > >> > > >> > https://lkml.org/lkml/2018/2/21/1202 > >> > > >> > Afterwards, on the OpenBSD side, it looks like one small change may be > >> > required in addition.. > >> > > >> > -Bryan. > >> > > >> > Index: sys/arch/amd64/amd64/identcpu.c > >> > =================================================================== > >> > RCS file: /cvs/src/sys/arch/amd64/amd64/identcpu.c,v > >> > retrieving revision 1.95.2.2 > >> > diff -u -p -u -r1.95.2.2 identcpu.c > >> > --- sys/arch/amd64/amd64/identcpu.c 30 Jul 2018 14:45:05 -0000 > >> > 1.95.2.2 > >> > +++ sys/arch/amd64/amd64/identcpu.c 1 Aug 2018 16:09:50 -0000 > >> > @@ -650,8 +650,10 @@ identifycpu(struct cpu_info *ci) > >> > > >> > msr = rdmsr(MSR_DE_CFG); > >> > #define DE_CFG_SERIALIZE_LFENCE (1 << 1) > >> > - msr |= DE_CFG_SERIALIZE_LFENCE; > >> > - wrmsr(MSR_DE_CFG, msr); > >> > + if ((msr & DE_CFG_SERIALIZE_LFENCE) == 0) { > >> > + msr |= DE_CFG_SERIALIZE_LFENCE; > >> > + wrmsr(MSR_DE_CFG, msr); > >> > + } > >> > } > >> > } > >> > > >> > > >> > >> As expected, -current works properly on real AMD hardware. So my assumption > >> about KVM doing something odd seems to be correct. > >> > >> The issue should be reported upstream to the KVM folks. But if the diff > >> above > >> also fixes the issue (I didn't test because I cannot reproduce it), ok > >> mlarkin. > >> > >> -ml > > > > I committed a fix for the potential MSR write #GP bug to -current: > > > > https://marc.info/?l=openbsd-cvs&m=153315564121057&w=2 > > > > Unfortunately, for the MSR read issue on older KVMs, it would require > > adding additional code to determine if we're running under KVM, there's > > really not much at all we can do here.. > > > > I agree these seem like KVM bugs, as this does not happen on real > > hardware, and at least also not in OpenBSD vmm(4). > > > > -Bryan. > > >
