>Synopsis: vmm boot order logic does not always work
>Category: amd64
>Environment:
System : OpenBSD 6.6
Details : OpenBSD 6.6 (GENERIC.MP) #3: Thu Nov 21 03:20:01 MST 2019
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64
Machine : amd64
>Description:
>How-To-Repeat:
>Fix:
I'm running 6.6-stable, and my vmm guest is failing to boot from cdrom
(despite a 'boot device cdrom' line in vm.conf) unless the vm has *two*
disks configured.
Configuration where guest does not boot (because only the cdrom is
bootable):
$ cat /etc/vm.conf
vm "test" {
disable
boot device cdrom
cdrom "/opt/vmmtest/bsd66.iso"
disk "/opt/vmmtest/var.img" format raw
#disk "/opt/vmmtest/opt.img" format raw
memory 1G
owner andrew
}
$ vmctl start -c test
Connected to /dev/ttyp7 (speed 115200)
No bootable device. Retrying in 60 seconds.
[EOT]
$ vmctl stop test
Configuration where guest does boot (from the cdrom):
$ cat /etc/vm.conf
vm "test" {
disable
boot device cdrom
cdrom "/opt/vmmtest/bsd66.iso"
disk "/opt/vmmtest/var.img" format raw
disk "/opt/vmmtest/opt.img" format raw
memory 1G
owner andrew
}
$ vmctl reload
$ vmctl start -c test
Connected to /dev/ttyp7 (speed 115200)
CD-ROM: E0
Loading /CDBOOT
probing: pc0 com0 mem[638K 1022M a20=on]
disk: hd0+* hd1+* cd0
>> OpenBSD/amd64 CDBOOT 3.44
boot>
cannot open cd0a:/etc/random.seed: No such file or directory
booting cd0a:/bsd.mp: 12739912+2937864+340000+0+704512
[993198+128+1013256+741007]=0x12940c8
entry point at 0xffffffff81001000
[ using 2748616 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
OpenBSD 6.6-stable (GENERIC.MP) #2: Sun Dec 15 20:33:06 NZDT 2019
..snip..
After digging through the vmd(8) code, the likely culprit was found
in fw_cfg.c, in the fw_cfg_init() function:
switch (vmc->vmc_bootdevice) {
case VMBOOTDEV_DISK:
bootorder = "/pci@i0cf8/*@2\nHALT";
break;
case VMBOOTDEV_CDROM:
bootorder = "/pci@i0cf8/*@4/*@0/*@0,100\nHALT";
break;
case VMBOOTDEV_NET:
/* XXX not yet */
bootorder = "HALT";
break;
}
Apparently in SeaBIOS-speak, the VMBOOTDEV_CDROM bootorder means
(roughly) "boot from target 0 lun 100 on the 4th PCI device". But
there's nothing in virtio.c that would ensure the cdrom is always
the 4th PCI device. So this bootorder is rather bogus to say the
least...
Even stranger, it's not even consistent across different hosts! I
have another host that will boot a guest from cdrom if there is *one*
disk configured, but not if there are *two* disks configured.
I'm not familiar enough with the vmd(8) internals to attempt a patch
for this (yet). Unlikely to be an easy fix, though.
Cheers
-Andrew
***
Below is the full output from the guest kernel -- note how the emulated
cdrom just so happens to end up at the right location on the PCI bus
because there were two emulated disks attached first:
***
Connected to /dev/ttyp7 (speed 115200)
CD-ROM: E0
Loading /CDBOOT
probing: pc0 com0 mem[638K 1022M a20=on]
disk: hd0+* hd1+* cd0
>> OpenBSD/amd64 CDBOOT 3.44
boot>
cannot open cd0a:/etc/random.seed: No such file or directory
booting cd0a:/bsd.mp: 12739912+2937864+340000+0+704512
[993198+128+1013256+741007]=0x12940c8
entry point at 0xffffffff81001000
[ using 2748616 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2019 OpenBSD. All rights reserved. https://www.OpenBSD.org
OpenBSD 6.6-stable (GENERIC.MP) #2: Sun Dec 15 20:33:06 NZDT 2019
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1056940032 (1007MB)
avail mem = 1012252672 (965MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf3f40 (10 entries)
bios0: vendor SeaBIOS version "1.11.0p2-OpenBSD-vmm" date 01/01/2011
bios0: OpenBSD VMM
acpi at bios0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz, 2593.52 MHz, 06-3a-09
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,LONG,LAHF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: OpenBSD
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio0
virtio0: irq 3
virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus1 at vioblk0: 2 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 2000MB, 512 bytes/sector, 4096000 sectors
virtio1: irq 5
virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk1 at virtio2
scsibus2 at vioblk1: 2 targets
sd1 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd1: 100000MB, 512 bytes/sector, 204800000 sectors
virtio2: irq 6
virtio3 at pci0 dev 4 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi0 at virtio3: qsize 128
scsibus3 at vioscsi0: 1 targets
cd0 at scsibus3 targ 0 lun 0: <OpenBSD, VMM CD-ROM, 001> removable
virtio3: irq 7
virtio4 at pci0 dev 5 function 0 "OpenBSD VMM Control" rev 0x00
vmmci0 at virtio4
virtio4: irq 9
isa0 at mainbus0
isadma0 at isa0
Local file?
com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
com0: console
vscsi0 at root
scsibus4 at vscsi0: 256 targets
softraid0 at root
scsibus5 at softraid0: 256 targets
root on cd0a swap on cd0b dump on cd0b
..snip..
***
Now comes the dmesg/usbdevs output from sendbug(1) on the host:
***
dmesg:
OpenBSD 6.6 (GENERIC.MP) #3: Thu Nov 21 03:20:01 MST 2019
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8472903680 (8080MB)
avail mem = 8203403264 (7823MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0000 (100 entries)
bios0: vendor Dell Inc. version "A21" date 05/08/2017
bios0: Dell Inc. Latitude E6430
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP APIC FPDT MCFG HPET SSDT SSDT SSDT DMAR ASF! SLIC BGRT
acpi0: wakeup devices P0P1(S4) USB1(S3) USB2(S3) USB3(S3) USB5(S3) USB6(S3)
USB7(S3) RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4)
PXSX(S4) RP07(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz, 2592.07 MHz, 06-3a-09
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1.2, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz, 2591.59 MHz, 06-3a-09
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xf8000000, bus 0-63
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (P0P1)
acpiprt2 at acpi0: bus 1 (RP01)
acpiprt3 at acpi0: bus 2 (RP02)
acpiprt4 at acpi0: bus -1 (RP05)
acpiprt5 at acpi0: bus 7 (RP06)
acpiprt6 at acpi0: bus -1 (RP07)
acpiprt7 at acpi0: bus -1 (RP08)
acpiprt8 at acpi0: bus -1 (PEG0)
acpiprt9 at acpi0: bus -1 (PEG1)
acpiprt10 at acpi0: bus -1 (PEG2)
acpiprt11 at acpi0: bus -1 (PEG3)
acpiprt12 at acpi0: bus -1 (RP03)
acpiprt13 at acpi0: bus 3 (RP04)
acpiec0 at acpi0
acpicpu0 at acpi0: C3(200@87 mwait.1@0x30), C2(500@59 mwait.1@0x10), C1(1000@1
mwait.1), PSS
acpicpu1 at acpi0: C3(200@87 mwait.1@0x30), C2(500@59 mwait.1@0x10), C1(1000@1
mwait.1), PSS
acpitz0 at acpi0: critical temperature is 107 degC
acpipci0 at acpi0 PCI0: 0x00000004 0x00000011 0x00000001
acpicmos0 at acpi0
"SMO8810" at acpi0 not configured
"*pnp0c14" at acpi0 not configured
acpibtn0 at acpi0: LID0
acpibtn1 at acpi0: PBTN
acpibtn2 at acpi0: SBTN
acpiac0 at acpi0: AC unit online
acpibat0 at acpi0: BAT0 model "DELL 05VFW2B" serial 2146 type LION oem
"LGC-LGC3.0"
acpibat1 at acpi0: BAT1 not present
acpibat2 at acpi0: BAT2 not present
"DELLABCE" at acpi0 not configured
acpivideo0 at acpi0: VID_
acpivout0 at acpivideo0: LCD_
cpu0: using VERW MDS workaround (except on vmm entry)
cpu0: Enhanced SpeedStep 2592 MHz: speeds: 2601, 2600, 2500, 2400, 2300, 2200,
2100, 2000, 1900, 1800, 1700, 1600, 1500, 1400, 1300, 1200 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 3G Host" rev 0x09
inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 4000" rev 0x09
drm0 at inteldrm0
inteldrm0: msi
xhci0 at pci0 dev 20 function 0 "Intel 7 Series xHCI" rev 0x04: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00
addr 1
"Intel 7 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
em0 at pci0 dev 25 function 0 "Intel 82579LM" rev 0x04: msi, address
e0:db:55:d2:7d:99
ehci0 at pci0 dev 26 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 16
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
addr 1
azalia0 at pci0 dev 27 function 0 "Intel 7 Series HD Audio" rev 0x04: msi
azalia0: codecs: IDT/0x76df, Intel/0x2806, using IDT/0x76df
audio0 at azalia0
ppb0 at pci0 dev 28 function 0 "Intel 7 Series PCIE" rev 0xc4: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 1 "Intel 7 Series PCIE" rev 0xc4: msi
pci2 at ppb1 bus 2
iwn0 at pci2 dev 0 function 0 "Intel Centrino Advanced-N 6205" rev 0x34: msi,
MIMO 2T2R, MoW, address 60:67:20:e9:2e:04
ppb2 at pci0 dev 28 function 3 "Intel 7 Series PCIE" rev 0xc4: msi
pci3 at ppb2 bus 3
ppb3 at pci0 dev 28 function 5 "Intel 7 Series PCIE" rev 0xc4: msi
pci4 at ppb3 bus 7
sdhc0 at pci4 dev 0 function 0 vendor "O2 Micro", unknown product 0x8221 rev
0x05: apic 2 int 17
sdhc0: SDHC 2.0, 50 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
ehci1 at pci0 dev 29 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 21
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
addr 1
pcib0 at pci0 dev 31 function 0 "Intel QM77 LPC" rev 0x04
ahci0 at pci0 dev 31 function 2 "Intel 7 Series AHCI" rev 0x04: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
ahci0: port 1: 1.5Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, INTEL SSDSC2CT12, 300i> naa.5001517387ebc654
sd0: 102400MB, 512 bytes/sector, 209715200 sectors, thin
cd0 at scsibus1 targ 1 lun 0: <HL-DT-ST, DVD+-RW GU70N, A102> removable
ichiic0 at pci0 dev 31 function 3 "Intel 7 Series SMBus" rev 0x04: apic 2 int 18
iic0 at ichiic0
iic0: addr 0x29 07=ff 0f=33 10=8a 11=0e 12=2a 13=52 14=b2 15=31 16=23 17=25
18=25 19=a8 1a=70 1b=64 1c=c0 1e=20 20=7f 22=40 27=ff 2b=02 2d=40 2f=20 30=95
31=25 32=0e 33=16 87=ff 8f=33 90=8a 91=0e 92=2a 93=52 94=b2 95=31 96=23 97=25
98=25 99=a8 9a=70 9b=64 9c=c0 9e=20 a0=7f a2=40 a7=ff a9=01 ad=41 af=20 b0=95
b1=25 b2=0e b3=16 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000
07=ffff
spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-12800 SO-DIMM
spdmem1 at iic0 addr 0x52: 4GB DDR3 SDRAM PC3-12800 SO-DIMM
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
efifb at mainbus0 not configured
umass0 at uhub0 port 6 configuration 1 interface 0 "Seagate Expansion" rev
3.00/1.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus2 at umass0: 2 targets, initiator 0
sd1 at scsibus2 targ 1 lun 0: <Seagate, Expansion, 9300>
sd1: 1907729MB, 512 bytes/sector, 3907029167 sectors
uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev
2.00/0.00 addr 2
uhub4 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev
2.00/0.00 addr 2
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (74e4035df7109b15.a) swap on sd0b dump on sd0b
inteldrm0: 1600x900, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation), using wskbd0
wsdisplay0: screen 1-5 added (std, vt100 emulation)
usbdevs:
Controller /dev/usb0:
addr 01: 8086:0000 Intel, xHCI root hub
super speed, self powered, config 1, rev 1.00
driver: uhub0
addr 02: 0bc2:2322 Seagate, Expansion
super speed, power 36 mA, config 1, rev 1.00, iSerial NAA8HBMV
driver: umass0
Controller /dev/usb1:
addr 01: 8086:0000 Intel, EHCI root hub
high speed, self powered, config 1, rev 1.00
driver: uhub1
addr 02: 8087:0024 Intel, Rate Matching Hub
high speed, self powered, config 1, rev 0.00
driver: uhub3
Controller /dev/usb2:
addr 01: 8086:0000 Intel, EHCI root hub
high speed, self powered, config 1, rev 1.00
driver: uhub2
addr 02: 8087:0024 Intel, Rate Matching Hub
high speed, self powered, config 1, rev 0.00
driver: uhub4
