On 11/13/15 12:22, P.A.M. van Dam (Pascal) wrote: > On Thu, Nov 12, 2015 at 07:17:49PM +0100, Laszlo Ersek wrote: > > Good afternoon all, > >> Please don't drop the list and other CC's from the discussion when the >> initial report is sent to the list. Keeping full context. > > My apologies for that. I noticed it when I got your personal out-of-office > reply. :) > > >> >> On 11/11/15 08:38, P.A.M. van Dam (Pascal) wrote: >>> On Mon, Nov 09, 2015 at 10:02:52PM +0100, Laszlo Ersek wrote: >>>> On 11/09/15 14:50, P.A.M. van Dam (Pascal) wrote: >>>>> >>>>> Good day all, >>>>> >>>>> I decided to do some testing with iSCSI boot from within a KVM. >>>>> >>>>> My setup is a Centos 7.1 KVM host with several Fedora 22 and Centos 7.1 >>>>> guests in it. The iSCSI storage has been configured >>>>> from a LenovoEMC EPX6300 NAS. >>>>> >>>>> 1. I've configured the iSCSI bootlun to be available for the OS install. >>>>> 2. Within the KVM guest OS install I assign the LUN to the Centos 7.1 OS. >>>>> 3. The OS installs without errors ont he iSCSI LUN. >>>>> >>>>> After reboot of the freshly installed OS on the KVM >>>>> I configure the iSCSI device in the OVMF user interface. >>>>> >>>>> 4. The iSCSI BOOT LUN is visible from the UEFI shell. >>>>> 5. When booting from the LUN, GRUB correcly displays the bootmenu. >>>> >>>> Am I to understand that GRUB (and its config) are correctly loaded via >>>> iSCSI? >>> >>> Yep. You are correct in this. GRUB loads without issue and both the menu >>> and commandline work. I can even access the kernel and the initrd from >>> GRUB's commandline. >>> >>>> >>>>> 6. When selecting and starting a kernel/initrd pair the started kernel >>>>> quickly crashes reporting a fault. >>>>> >>>>> My questions; >>>>> >>>>> 1. Is iSCSI boot support in the OVMF/EDKII currrently usable? >>> >>> >>>> >>>> I never tested it, but I guess it should work (for some value of >>>> "work"). Since I never tested it, I can't say what code in OvmfPkg would >>>> require specific support (perhaps the boot order library... dunno). >>>> >>>>> 2. Can I help in further analyzing the issue? It could be an issue in >>>>> EDKII (drivers?), in grub or in the Linux kernel. >>>> >>>> Your email is missing almost all of the important details, so let's >>>> start again. :) >>> >>> I will try to supply the info needed. This is my first time on this >>> mailinglist and it's long time since I've joined a Linux project. >>> >>>> >>>> - Are you using libvirt? If so, what version? What is your domain XML? >>>> >>> >>> Yes, I am using libvirt. I am currently using >>> libvirt-1.2.8-16.el7_1.5.x86_64. >>> >>> [virtlabn17.xml] >>> >>> [root@kallista:38 qemu]# cat virtlabn17.xml >>> <!-- >>> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE >>> OVERWRITTEN AND LOST. Changes to this xml configuration should be made >>> using: >>> virsh edit virtlabn17 >>> or other application using the libvirt API. >>> --> >>> >>> <domain type='kvm'> >>> <name>virtlabn17</name> >>> <uuid>ab2b120f-8a4f-4f90-bd32-fa78fedd51b0</uuid> >>> <description>Centos 7.1 UEFI/ISCSI BOOT virtlabn17</description> >>> <memory unit='KiB'>2097152</memory> >>> <currentMemory unit='KiB'>2097152</currentMemory> >> >> Okay, so 2GB RAM. >> >>> <vcpu placement='static'>2</vcpu> >> >> 2 VCPUs >> >>> <os> >>> <type arch='x86_64' machine='pc-i440fx-rhel7.1.0'>hvm</type> >>> <loader readonly='yes' >>> type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> >>> <nvram >>> template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'>/var/lib/libvirt/qemu/nvram/virtlabn17_VARS.fd</nvram> >> >> Looks good. The OVMF binary is from Gerd. > > Correct. > >> >>> <boot dev='hd'/> >> >> This is kinda useless here (you don't have a disk in your config). In >> any case, as a side point, the <boot order='N'/> elements within device >> elements are much more flexible than this. See >> <http://libvirt.org/formatdomain.html#elementsNICSBoot>, and then search >> that page for more occurrences of the string "boot order=". See also the >> "boot" element's docs at >> <http://libvirt.org/formatdomain.html#elementsOSBIOS>. > > Ok. I will take a look at it. But as far as I know, it's not the cause of the > issue. :) >> >>> <bootmenu enable='yes'/> >> >> This will drop you to the edk2 setup page; I guess it's fine. > > Correct. > >> >>> </os> >>> <features> >>> <acpi/> >>> <apic/> >>> <pae/> >>> </features> >>> <cpu mode='custom' match='exact'> >>> <model fallback='allow'>SandyBridge</model> >>> </cpu> >>> <clock offset='utc'> >>> <timer name='rtc' tickpolicy='catchup'/> >>> <timer name='pit' tickpolicy='delay'/> >>> <timer name='hpet' present='no'/> >>> </clock> >>> <on_poweroff>destroy</on_poweroff> >>> <on_reboot>restart</on_reboot> >>> <on_crash>restart</on_crash> >>> <devices> >>> <emulator>/usr/libexec/qemu-kvm</emulator> >>> <controller type='usb' index='0' model='ich9-ehci1'> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' >>> function='0x7'/> >>> </controller> >>> <controller type='usb' index='0' model='ich9-uhci1'> >>> <master startport='0'/> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' >>> function='0x0' multifunction='on'/> >>> </controller> >>> <controller type='usb' index='0' model='ich9-uhci2'> >>> <master startport='2'/> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' >>> function='0x1'/> >>> </controller> >>> <controller type='usb' index='0' model='ich9-uhci3'> >>> <master startport='4'/> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' >>> function='0x2'/> >>> </controller> >>> <controller type='pci' index='0' model='pci-root'/> >>> <controller type='virtio-serial' index='0'> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' >>> function='0x0'/> >>> </controller> >>> <interface type='bridge'> >>> <mac address='52:54:00:0e:2c:f1'/> >>> <source bridge='br0'/> >>> <model type='virtio'/> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' >>> function='0x0'/> >>> </interface> >> >> I see. This is the "Bridge to LAN" config >> <http://libvirt.org/formatdomain.html#elementsNICSBridge>. >> >> You are using virtio-net, and the iPXE oprom will be present in the PCI >> ROM BAR. >> >> Idea (1): not that it might matter a lot, but you could try with the >> builtin virtio-net driver. For that, add the following element within >> <interface>: >> >> <rom bar='off'/> >> > > Yes. I implemented Idea(1) and it works! So there definately is something > with this ROM that cause the > crash. (Looking further in the email, not the crash of the kernel, but > earlier in the process) > > So; conclusion for Idea(1); it's a solution. The system boots fine. It spits > out some udev messages about > the dubious use of devices in the kernel namespace realm vs. userpsace realm, > but it works > Thanks. > > I also tried Idea(2). > >>> <serial type='pty'> >>> <target port='0'/> >>> </serial> >>> <console type='pty'> >>> <target type='serial' port='0'/> >>> </console> >>> <input type='tablet' bus='usb'/> >>> <sound model='ich6'> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' >>> function='0x0'/> >>> </sound> >>> <redirdev bus='usb' type='spicevmc'> >>> </redirdev> >>> <redirdev bus='usb' type='spicevmc'> >>> </redirdev> >>> <memballoon model='virtio'> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' >>> function='0x0'/> >>> </memballoon> >>> </devices> >>> </domain> >>> >>> [/virtlabn17.xml] >>> >>> >>>> - What version of QEMU are you using? What is your command line? >>>> >>> >>> /usr/libexec/qemu-kvm --version >>> QEMU emulator version 2.1.2 (qemu-kvm-ev-2.1.2-23.el7_1.8.1), Copyright (c) >>> 2003-2008 Fabrice Bellard >>> >>> Commandline appears to be (as invoked by virsh) >>> >>> /usr/libexec/qemu-kvm -name virtlabn17 -S -machine >>> pc-i440fx-rhel7.1.0,accel=kvm,usb=off -cpu SandyBridge -drive >>> file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on >>> -drive >>> file=/var/lib/libvirt/qemu/nvram/virtlabn17_VARS.fd,if=pflash,format=raw,unit=1 >>> -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid >>> ab2b120f-8a4f-4f90-bd32-fa78fedd51b0 -nographic -no-user-config >>> -nodefaults -chardev >>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/virtlabn17.monitor,server,nowait >>> -mon chardev=charmonitor,id=monitor,mode=control -rtc >>> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard >>> -no-hpet -no-shutdown -boot order=c,menu=on,strict=on -device >>> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device >>> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 >>> -device >>> ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 >>> -device >>> ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 >>> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 >>> -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=62 -device >>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0e:2c:f1,bus=pci.0,addr=0x3 >>> -chardev pty,id=charserial0 -device >>> isa-serial,chardev=charserial0,id=serial0 -device >>> usb-tablet,id=input0 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 >>> -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev >>> spicevmc,id=charredir0,name=usbredir -device >>> usb-redir,chardev=charredir0,id=redir0 -chardev >>> spicevmc,id=charredir1,name=usbredir -device >>> usb-redir,chardev=charredir1,id=redir1 -device >>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on >>> >>> >>> >>>> - How did you build OVMF? Do you have a verbose debug log from the OVMF >>>> run? >>> >>> I got OVMF from the nightly builds in the Krax repos: >>> >>> edk2.git-ovmf-x64-0-20151110.b1308.g9ddd7d7.noarch >> >> Right. It looks fresh. >> > > Updated every day for my testing purposes. > >>> >>>> >>>> - I could probably derive the answer to the next question from the >>>> above, but I'll ask anyway: are you using iPXE (bundled with QEMU) >>>> for driving your virtual NIC (utilized by edk2 and grub), or OVMF's >>>> builtin virtio-net driver? If you don't know (or use a NIC model >>>> different from virtio-net), then the answer is iPXE. >>> >>> iPXE is reporting. But KVM is using virtio-net. >> >> Right. >> >>> >>>> >>>> - What is the complete (= "ignore_loglevel") guest kernel log, up to and >>>> including the crash? >>> >>> GRUB commandline is: >>> >>> linuxefi /vmlinuz-3.10.0-229.el7.x86_64 root=UUID=66f3dbf7-78e9-4c10-8\ >>> 702-5ae04159a2f3 ro >>> netroot=iscsi:@10.8.63.157::3260::iqn.2012-07.com.lenovoem\ >>> c:storage.epx6300n2.n2iscsiv05 crashkernel=auto >>> ifname=eth0:52:54:00:0e:2c:f1 \ >>> iscsi_initiator=iqn.1994-05.com.redhat:virtlabn17 ip=eth0:dhcp nomodeset >>> ignor\ >>> e_loglevel >>> >>> initrdefi /initramfs-3.10.0-229.el7.x86_64.img >>> >>> >>> >>> And the kernel log is not that much, so I can include it here. It dies very >>> early in the boot process. >>> >>> !!!! X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID - >>> 00000000 !!!! >>> RIP - 000000007FF67FDA, CS - 0000000000000038, RFLAGS - 0000000000210286 >>> ExceptionData - 0000000000000000 >>> RAX - AFAFAFAFAFAFAFAF, RCX - AFAFAFAFAFAFAFAF, RDX - 000000007E984630 >>> RBX - 000000007E85C628, RSP - 000000007FF58418, RBP - 000000007FF58430 >>> RSI - 000000007F04C898, RDI - AFAFAFAFAFAFAFAF >>> R8 - 000000007F263818, R9 - 000000007F04AC58, R10 - 000000007F0B6898 >>> R11 - 000000007E863000, R12 - 000000007E85BAD0, R13 - 000000007FF59768 >>> R14 - 000000007FF59778, R15 - 0000000000000000 >>> DS - 0000000000000030, ES - 0000000000000030, FS - 0000000000000030 >>> GS - 0000000000000030, SS - 0000000000000030 >>> CR0 - 0000000080000033, CR2 - 0000000000000000, CR3 - 000000007FEF8000 >>> CR4 - 0000000000000668, CR8 - 0000000000000000 >>> DR0 - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000 >>> DR3 - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400 >>> GDTR - 000000007FEE67D8 0000000000000047, LDTR - 0000000000000000 >>> IDTR - 000000007F69D018 0000000000000FFF, TR - 0000000000000000 >>> FXSAVE_STATE - 000000007FF58070 >>> !!!! Find PE image >>> /home/jenkins/workspace/edk2/rpmbuild/rpm/BUILD/edk2.git/Build/OvmfX64/DEBUG_GCC49/X64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll >>> (ImageBase=000000007FF5A000, EntryPoint=000000007FF5A240) !!!! >>> >>> Hope this helps. >> >> It does. >> >> This is not a guest kernel log. It is a register dump, due to a GP >> fault, from the edk2 DXE_CORE that is built into the OVMF binary. With >> the error message printed, I could locate the exact source code point, >> but for that I'd need the Build subdirectory (with the intermediate >> build files) from Gerd's build system -- and that subdirectory doesn't >> exist any longer. >> >> You wrote: >> >>> 5. When booting from the LUN, GRUB correcly displays the bootmenu. >>> 6. When selecting and starting a kernel/initrd pair the started >>> kernel quickly crashes reporting a fault. >> >> The above means that you don't reach the kernel (well, not those parts >> of it that would run after the EFI stub's ExitBootServices() call). > > That makes sense to me. > >> >> I wonder what could cause the DXE_CORE to blow up. I suspect it happens >> near (or in) gBS->ExitBootServices(). >> >> That UEFI boot service is implemented with the CoreExitBootServices() >> function in "MdeModulePkg/Core/Dxe/DxeMain/DxeMain.c", and UEFI drivers >> for NICs are strongly recommended to abort pending transfers / re-set >> cards in EVT_SIGNAL_EXIT_BOOT_SERVICES callbacks. >> >> ... The iPXE packages on CentOS's Community Build System >> <http://cbs.centos.org/koji/packageinfo?packageID=87> look rather old; >> the most recent build seems to be "ipxe-20130517-7.gitc4bce43.el7". (The >> virtio-net option ROM in question comes from the "ipxe-roms-qemu" >> subpackage.) >> >> In 2015 there have been fixes in iPXE that could affect your booting. >> For example, 755d2b8f ("Ensure drivers are disconnected when >> ExitBootServices() is called"). >> >> Idea (2): can you please install the "ipxe.git-combined" package from >> Gerd's repo as well, and then add the following to your <interface> >> element (note: this is mutually exclusive with idea (1)): >> >> <rom bar='on' file='/usr/share/ipxe.git/combined/1af41000.rom'/> >> > > I implemented this also (removing the rom bar='off' first ofcourse) > and this leads to the same crash and register dump. Ergo, both entities > definately bite eachother. > > Looking at it one way, my issue is solved now. I can boot from my iSCSI LUNs > and will > conduct further testing with booting from DM MULTIPATH devices etc. > > But... are there scenario's possible where one would like to have both > the combined ROM and iSCSI booting?
Yes, such scenarios are possible. For example, if you use a NIC emulated by QEMU for which OVMF has no builtin driver (like "rtl8139"), but iPXE does. (Please refer to the "Network Support" section in OvmfPkg/README.) In those cases the iPXE oprom is your only option. I suggest to discuss this further with Michael Brown, the iPXE maintainer (CC'd). Thanks Laszlo > And so, should this be solved? Or.. > are there any other case that could trigger this behaviour? > > Thanks for your help. Happily testing further. > > If you need more testcases executed; please say so. > > Kind regards, > > Pascal > > > >> Thanks >> Laszlo >> >>> >>> Glad to be of assistance in testing this. Having the KVMs boot from >>> UEFI/ISCSI targets would be a great benefit. I will document my >>> findings and write a recipe how to implement it for the project. >>> >>> >>>> >>>> CC'ing Gary (I seem to recall that Gary has experience with iSCSI >>>> booting under OVMF -- see >>>> <https://github.com/tianocore/edk2/commit/36c6413f>.) >>>> >>>> Thanks >>>> Laszlo >>>> >>>>> >>>>> Thanks for your time. >>>>> >>>>> Kind regards, >>>>> >>>>> Pascal >>>>> >>>>> _______________________________________________ >>>>> edk2-devel mailing list >>>>> edk2-devel@lists.01.org >>>>> https://lists.01.org/mailman/listinfo/edk2-devel >>>>> >>>> >>>> _______________________________________________ >>>> edk2-devel mailing list >>>> edk2-devel@lists.01.org >>>> https://lists.01.org/mailman/listinfo/edk2-devel >> >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
