Bug#869807: qemu: [regression] unknown CPU feature __kvm_hv_spinlocks after upgrade to +deb9u1
Hi, On 08/05/2017 02:06 PM, Michael Tokarev wrote: > 05.08.2017 15:02, Christian Seiler wrote: xhci-guard-xhci_kick_epctx-against-recursive-calls-CVE-2017-9375.patch >>> >>> What's the complete qemu command line? >> >> It's quite long (generated from libvirt), I posted that in the initial >> bug report: >> >> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=869807#5 >> >>> Does it include nec-xhci? >> >> Yes, it does: >> >> -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x5 >> >> (I did configure XHCI in libvirt to be able to pass through >> USB 3 devices.) >> >> And indeed, if I change that back to USB 2.0 in libvirt's configuration, >> and install +deb9u1, the VM now boots again. > > This is #869945, and the actual problem is not what you've > posted but the xhci assertion failure. Yeah, I just noticed that the __kvm_hv_spinlocks error also occurs if the VM is started successfully - and is in fact only a warning message. Unfortunately, due to the way libvirtd invokes qemu I don't see any assertion failure in the logs (or I just don't know which log to look at). > I'll merge this bug with #869945. Yes, I can verify that the VM boots again with +deb9u2. (I had kept the package on hold via apt-mark until I knew the problem was resolved.) Sorry for providing a red herring with the CPU flag. And many thanks for your work! Regards, Christian
Bug#869807: qemu: [regression] unknown CPU feature __kvm_hv_spinlocks after upgrade to +deb9u1
05.08.2017 15:02, Christian Seiler wrote: >>> xhci-guard-xhci_kick_epctx-against-recursive-calls-CVE-2017-9375.patch >> >> What's the complete qemu command line? > > It's quite long (generated from libvirt), I posted that in the initial > bug report: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=869807#5 > >> Does it include nec-xhci? > > Yes, it does: > > -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x5 > > (I did configure XHCI in libvirt to be able to pass through > USB 3 devices.) > > And indeed, if I change that back to USB 2.0 in libvirt's configuration, > and install +deb9u1, the VM now boots again. This is #869945, and the actual problem is not what you've posted but the xhci assertion failure. I'll merge this bug with #869945. Thanks, /mjt
Bug#869807: qemu: [regression] unknown CPU feature __kvm_hv_spinlocks after upgrade to +deb9u1
Hi, On 08/05/2017 01:55 PM, Michael Tokarev wrote: > 05.08.2017 11:48, Christian Seiler wrote: > >> xhci-guard-xhci_kick_epctx-against-recursive-calls-CVE-2017-9375.patch > > What's the complete qemu command line? It's quite long (generated from libvirt), I posted that in the initial bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=869807#5 > Does it include nec-xhci? Yes, it does: -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x5 (I did configure XHCI in libvirt to be able to pass through USB 3 devices.) And indeed, if I change that back to USB 2.0 in libvirt's configuration, and install +deb9u1, the VM now boots again. Regards, Christian
Bug#869807: qemu: [regression] unknown CPU feature __kvm_hv_spinlocks after upgrade to +deb9u1
05.08.2017 11:48, Christian Seiler wrote: > xhci-guard-xhci_kick_epctx-against-recursive-calls-CVE-2017-9375.patch What's the complete qemu command line? Does it include nec-xhci? /mjt
Bug#869807: qemu: [regression] unknown CPU feature __kvm_hv_spinlocks after upgrade to +deb9u1
Hi, On 08/04/2017 01:56 AM, Christian Seiler wrote: > Any ideas? My guess would be to selectively enable different > patches in debian/patches/series and try to figure out which > once of these actually causes the issue. (Could take me a > while though.) Or do you have any better suggestions? I've now built qemu 7 times, selectively enabling only one of the patches in +deb9u1 each time. (I noticed in git that you already noticed yourself that the 8th patch mentioned in the changelog was not included in d/patches/series.) I found the culprit: all variants boot my system except for the one where this patch is enabled, which precisely produces the symptoms described in this bug. xhci-guard-xhci_kick_epctx-against-recursive-calls-CVE-2017-9375.patch https://anonscm.debian.org/cgit/pkg-qemu/qemu.git/tree/debian/patches/xhci-guard-xhci_kick_epctx-against-recursive-calls-CVE-2017-9375.patch?id=73d4f3ee4989908f1fc00bc82972352d59f3c688 But looking at the patch: I really don't get it. It appears to have nothing to do with CPU flags at all. Any ideas? I'm not very familiar with qemu's source code, but am quite comfortable with a debugger. If you have any idea on how I could investigate this further... Regards, Christian
Bug#869807: qemu: [regression] unknown CPU feature __kvm_hv_spinlocks after upgrade to +deb9u1
Hi, On 08/02/2017 11:58 PM, Christian Seiler wrote: > - first of all I'll try to upgrade qemu and then reboot >the entire system before restarting the VM (to make sure >it's not some caching issue you mentioned) Rebooting after upgrading qemu doesn't change anything. > - recompile both 1:2.8+dfsg-6 and 1:2.8+dfsg-6+deb9u1 >in a clean Stretch build environment and see if I can >reproduce the issue with self-built packages I've recompiled both 1:2.8+dfsg-6 and 1:2.8+dfsg-6+deb9u1 in a clean stretch sbuild environment (I can provide sbuild logs if necessary) and I get the same results: - the version without +deb9u1 works - the version with +deb9u1 has precisely this bug This is really weird. I even debdiff'd the source packages (to see if perhaps the uploaded package didn't correspond to the git repo, but the source package in the archive is fine) - and I really don't understand it: nothing in the debdiff touches any file that has to do with CPU features, as far as I can tell... Any ideas? My guess would be to selectively enable different patches in debian/patches/series and try to figure out which once of these actually causes the issue. (Could take me a while though.) Or do you have any better suggestions? Regards, Christian
Bug#869807: qemu: [regression] unknown CPU feature __kvm_hv_spinlocks after upgrade to +deb9u1
Hi, Thanks for your response! On 08/02/2017 11:42 PM, Michael Tokarev wrote: > The thing is that there are no changes in +deb9u1 which > can lead to anything like that, at all. Yes, I noticed that myself, which makes it really weird. > Unless, which is > also a possibility, there's a bug in my build environment > (I use regular sbuild stretch chroot for that). Does qemu embed anything from its Build-Dependencies so that the change was perhaps not in qemu but in something else and +deb9u1 just happened to pick it up because it was compiled later? > I've no idea what can cause this, and where this 'feature' > come from to start with - it smells like some libvirt thing, > but I'm not sure. Well, to be honest I was surprised to see all of these CPU flags specified by libvirt. I would have expected it to just have Skylake-Client as the cpu, plus potentially one or two flags to support the IOMMU passthrough of devices, but not these many. > But your environment is quite a bit unusual, In some sense yes. But in some other sense it isn't that weird. I set it up in the following way: - Stretch system (while it was still frozen but close to the release) - used virt-manager to create a new VM. Selected Windows 10 as OS, then before installing the OS I made sure that the processor was not the Host CPU, but a Skylake (by selecting it from the CPU list provided by libvirt); I did this to make sure I could at a later point in time migrate the VM to a different system without having to reactive Windows - after installing the QEMU Guest Agent on the OS I changed the hard disks + NIC to use virtio (for performance) - later on after installing the aforementioned PCIe card in the computer itself, I just added a PCI Host Device in virt-manager to the VM image That causes libvirt to run qemu with all of these options. > Did you try to restart libvirtd to start with If I remember correctly I did a reboot of the computer to check that the BIOS settings were still all OK. (i.e. VT-x and VT-d enabled) But I'm not completely sure about that. I might not get to that before Monday, but I'll try the following next: - first of all I'll try to upgrade qemu and then reboot the entire system before restarting the VM (to make sure it's not some caching issue you mentioned) - recompile both 1:2.8+dfsg-6 and 1:2.8+dfsg-6+deb9u1 in a clean Stretch build environment and see if I can reproduce the issue with self-built packages I'll update this bug report once I know more. Regards, Christian
Bug#869807: qemu: [regression] unknown CPU feature __kvm_hv_spinlocks after upgrade to +deb9u1
26.07.2017 18:49, Christian Seiler wrote: > Package: qemu-system-x86 > Version: 1:2.8+dfsg-6+deb9u1 > Severity: important > X-Debbugs-Cc: secur...@debian.org > > Dear maintainers, dear security team, > > after performing the security upgrade to 1:2.8+dfsg-6+deb9u1 a virtual > machine (managed via libvirt) does not start anymore. > > Underlying CPU is an Intel Kaby Lake Core i7-7700. VT-x and VT-d are > enabled in the BIOS. Kernel cmdline has intel_iommu=on. Latest > microcode update is installed. > > KVM configuration: Machine of guest is set to pc-i440fx-2.8, CPU is set > to Skylake-Client. A PCIe framegrabber card (in x16 slot, but card is > x4 or x8, I don't remember exactly) is passed through to the guest. > > With 1:2.8+dfsg-6 the guest boots just fine. > > With 1:2.8+dfsg-6+deb9u1 the guest doesn't start properly. In the > journal I can find the following message every time I try to start the > guest: > > libvirtd[964]: ...: 984: error : x86FeatureInData:780 : internal error: > unknown CPU feature __kvm_hv_spinlocks > libvirtd[964]: ...: 964: error : qemuMonitorIO:695 : internal error: End of > file from qemu monitor > > To get this working again I downgraded qemu-kvm, qemu-system-common > and qemu-system-x86 back to 1:2.8+dfsg-6. Only qemu-system-x86 is relevant here, the rest is not. The thing is that there are no changes in +deb9u1 which can lead to anything like that, at all. Unless, which is also a possibility, there's a bug in my build environment (I use regular sbuild stretch chroot for that). I've no idea what can cause this, and where this 'feature' come from to start with - it smells like some libvirt thing, but I'm not sure. At least there's no such string in qemu itself, there is, however, hv-spinlocks feature in qemu, and it definitely hasn't changed in +deb9u1. Hmm.. But your environment is quite a bit unusual, I highly doubt I for one will be able to replicate it. Did you try to restart libvirtd to start with - maybe some cached data went out of sync after upgrade? Thanks, /mjt
Bug#869807: qemu: [regression] unknown CPU feature __kvm_hv_spinlocks after upgrade to +deb9u1
Package: qemu-system-x86 Version: 1:2.8+dfsg-6+deb9u1 Severity: important X-Debbugs-Cc: secur...@debian.org Dear maintainers, dear security team, after performing the security upgrade to 1:2.8+dfsg-6+deb9u1 a virtual machine (managed via libvirt) does not start anymore. Underlying CPU is an Intel Kaby Lake Core i7-7700. VT-x and VT-d are enabled in the BIOS. Kernel cmdline has intel_iommu=on. Latest microcode update is installed. KVM configuration: Machine of guest is set to pc-i440fx-2.8, CPU is set to Skylake-Client. A PCIe framegrabber card (in x16 slot, but card is x4 or x8, I don't remember exactly) is passed through to the guest. With 1:2.8+dfsg-6 the guest boots just fine. With 1:2.8+dfsg-6+deb9u1 the guest doesn't start properly. In the journal I can find the following message every time I try to start the guest: libvirtd[964]: ...: 984: error : x86FeatureInData:780 : internal error: unknown CPU feature __kvm_hv_spinlocks libvirtd[964]: ...: 964: error : qemuMonitorIO:695 : internal error: End of file from qemu monitor To get this working again I downgraded qemu-kvm, qemu-system-common and qemu-system-x86 back to 1:2.8+dfsg-6. The full command that is used to start qemu by libvirt is the following (UUIDs and MAC addresses censored): qemu-system-x86_64 -enable-kvm -name guest=win10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-win10/master-key.aes -machine pc-i440fx-2.8,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+pdpe1gb,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -m 16384 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-win10/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/win10.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/libvirt/images/win10_data.qcow2,format=qcow2,if=none,id=drive-virtio-disk1 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,id=drive-ide0-0-1,readonly=on -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=25,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=...,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0xa -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on I did take a look at the patches in the git repository: https://anonscm.debian.org/cgit/pkg-qemu/qemu.git/log/?id=refs/heads/debian-stretch But I'm very confused because none of these patches actually touch the CPU flags or any other part of virtualization. Regards, Christian -- System Information: Debian Release: 9.1 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.9.0-3-amd64 (SMP w/8 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages qemu-system-x86 depends on: ii ipxe-qemu 1.0.0+git-20161027.b991c67-1 ii libaio1 0.3.110-3 ii libasound2 1.1.3-5 ii libbluetooth3 5.43-2 ii libbrlapi0.65.4-7 ii libc6 2.24-11+deb9u1 ii libcacard0 1:2.5.0-3 ii libfdt1 1.4.2-1 ii libgcc1 1:6.3.0-18 ii libglib2.0-02.50.3-2 ii libgnutls30 3.5.8-5+deb9u2 ii libjpeg62-turbo 1:1.5.1-2 ii libncursesw56.0+20161126-1 ii libnettle6 3.3-1+b1 ii libnuma1
