>Synopsis: cannot clean-install KVM/QEMU VM that don't support MSR_TSX_CTRL
>Category: kernel
>Environment:
System : OpenBSD 6.7
Details : OpenBSD 6.7 (GENERIC.MP) #2: Thu May 21 18:28:46 JST 2020
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64
Machine : amd64
>Description:
cpu_tsx_disable() in sys/arch/amd64/amd64/cpu.c tries to set
MSR_TSX_CTRL register, there is no problem with "real" CPU.
But under KVM/QEMU, OpenBSD-6.7 will crash if they don't handle
that MSR register.
There is no way to by-pass cpu_tsx_disable(), we cannot run
official binary on old KVM/QEMU host.
>How-To-Repeat:
simply try to boot OpenBSD-6.7/amd64's bsd.rd, bsd.sp and bsd.mp
>Fix:
update KVM/QEMU. otherwise upgrade from OpenBSD-6.6 and following
workaround procedure.
1) install OpenBSD-6.6
2) fetch OpenBSD-6.7 kernel source code
3) modify sys/arch/amd64/amd64/cpu.c to disable cpu_tsx_disable()
4) build OpenBSD-6.7 GENERIC.MP kernel (as bsd.mp.tmp)
5) build OpenBSD-6.7 RAMDISK_CD kernel (as bsd.rd.tmp)
6) add OpenBSD-6.7 installer from bsd.rd to bsd.rd.tmp by rdsetroot
7) boot bsd.rd.tmp and upgrade
8) boot bsd.mp.tmp
9) rebuild OpenBSD-6.7 GENERIC.MP kernel and install
attached dmesg is modified kernel, by this patch
--- cpu.c~ Thu May 21 20:27:36 2020
+++ cpu.c Thu May 21 18:12:58 2020
@@ -1175,9 +1175,10 @@
(sefflags_edx & SEFF0EDX_ARCH_CAP)) {
msr = rdmsr(MSR_ARCH_CAPABILITIES);
if (msr & ARCH_CAPABILITIES_TSX_CTRL) {
- msr = rdmsr(MSR_TSX_CTRL);
- msr |= TSX_CTRL_RTM_DISABLE | TSX_CTRL_TSX_CPUID_CLEAR;
- wrmsr(MSR_TSX_CTRL, msr);
+ printf("%s: modifying MSR_TSX_CTRL bypassed\n",
ci->ci_dev->dv_xname);
+// msr = rdmsr(MSR_TSX_CTRL);
+// msr |= TSX_CTRL_RTM_DISABLE | TSX_CTRL_TSX_CPUID_CLEAR;
+// wrmsr(MSR_TSX_CTRL, msr);
}
}
}
dmesg:
OpenBSD 6.7 (GENERIC.MP) #2: Thu May 21 18:28:46 JST 2020
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1056940032 (1007MB)
avail mem = 1012346880 (965MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf6200 (11 entries)
bios0: vendor Seabios version "0.5.1" date 01/01/2011
bios0: Red Hat KVM
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S5
acpi0: tables DSDT FACP SSDT APIC
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz, 344.08 MHz, 06-55-07
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line
16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: modifying MSR_TSX_CTRL bypassed
cpu1: Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz, 673.84 MHz, 06-55-07
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line
16-way L2 cache
cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: smt 0, core 0, package 1
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"QEMU0002" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0
wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 1.5.> removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
iic0: addr 0x18 00=0c 01=91 02=91 03=91 04=91 05=91 06=91 07=91 08=91 09=85
0a=e2 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff
04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x19 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1a 00=0c 01=91 02=91 03=91 04=91 05=91 06=91 07=91 08=91 09=85
0a=e2 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff
04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1b 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1c 0f=06 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1d 0f=06 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1e 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1f 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x20 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x21 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x22 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x23 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x24 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x25 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x26 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x27 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x28 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x29 00=0c 01=91 02=91 03=91 04=91 05=91 06=91 07=91 08=91 09=85
0a=e2 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff
03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2a 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2b 00=0c 01=91 02=91 03=91 04=91 05=91 06=91 07=91 08=91 09=85
0a=e2 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff
03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2c 00=0c 0e=0c 16=0c 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c
words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2d 00=0c 04=0c 0e=0c 16=0c 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c
fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2e 00=0c 04=0c 0e=0c 16=0c 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c
fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2f 00=0c 0e=0c 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words
00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x48 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x49 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4a 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4b 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4c 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=0c
48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff
05=ffff 06=ffff 07=ffff
iic0: addr 0x4d 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=0c
48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff
05=ffff 06=ffff 07=ffff
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 9c:a3:ba:02:00:3c
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00
vio1 at virtio1: address 9c:a3:ba:03:86:dc
virtio1: msix shared
virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Network" rev 0x00
vio2 at virtio2: address 9c:a3:ba:05:0d:7c
virtio2: msix shared
virtio3 at pci0 dev 6 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio3
scsibus2 at vioblk0: 2 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 102400MB, 512 bytes/sector, 209715200 sectors
virtio3: msix shared
virtio4 at pci0 dev 7 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
viomb0 at virtio4
virtio4: apic 0 int 11
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00
addr 1
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev
2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (a279a6bea6e758ae.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 1: density unknown
usbdevs:
Controller /dev/usb0:
addr 01: 8086:0000 Intel, UHCI root hub
full speed, self powered, config 1, rev 1.00
driver: uhub0
addr 02: 0627:0001 QEMU, QEMU USB Tablet
full speed, power 100 mA, config 1, rev 0.00, iSerial 42
driver: uhidev0
