[Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
** Also affects: qemu-gitlab via https://gitlab.com/qemu-project/qemu/-/issues/337 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1876678 Title: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve To manage notifications about this bug go to: https://bugs.launchpad.net/qemu-gitlab/+bug/1876678/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Hi Christian, I have posted issue to upstream QEMU https://gitlab.com/qemu-project/qemu/-/issues/337 Thanks again for assistance / advise. Cheers from Oz, John. ** Bug watch added: gitlab.com/qemu-project/qemu/-/issues #337 https://gitlab.com/qemu-project/qemu/-/issues/337 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1876678 Title: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1876678/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
On Mon, May 17, 2021 at 3:01 PM John Hartley <1876...@bugs.launchpad.net> wrote: > > Hi Christian, > > now I have env setup that test is pretty straight forward. Glad to hear that1 > $ qemu-system-x86_64 --version > QEMU emulator version 6.0.50 (v6.0.0-540-g6005ee07c3) ... > > So looks like an upstream candidate. Yeah failing with this (latest release and no Ubuntu Delta applied) certainly unlocks you to report it there. As I said, once you do so it would be great to add a link here pointing to the issue you've filed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1876678 Title: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1876678/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Hi Christian,
now I have env setup that test is pretty straight forward.
Here are results:
Build QEMU with following configuration:
../configure --disable-werror --disable-user --disable-linux-user
--disable-docs --disable-guest-agent --disable-sdl --disable-gtk
--disable-vnc --disable-xen --disable-brlapi --disable-fdt --disable-hax
--disable-vde --disable-netmap --disable-rbd --disable-libiscsi
--disable-libnfs --disable-smartcard --disable-libusb --disable-usb-
redir --disable-seccomp --disable-glusterfs --disable-tpm --disable-numa
--disable-opengl --disable-virglrenderer --disable-xfsctl --disable-
slirp --disable-blobs --target-list=x86_64-softmmu --disable-rdma
--disable-pvrdma --disable-attr --disable-vhost-net --disable-vhost-
vsock --disable-vhost-scsi --disable-vhost-crypto --disable-vhost-user
--disable-spice --disable-qom-cast-debug --disable-bochs --disable-cloop
--disable-dmg --disable-qcow1 --disable-vdi --disable-vvfat --disable-
qed --disable-parallels --disable-avx2 --disable-nettle --disable-gnutls
--disable-capstone --enable-tools
Version of QEMU SYSTEM:
$ qemu-system-x86_64 --version
QEMU emulator version 6.0.50 (v6.0.0-540-g6005ee07c3)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
Result when running Layer 2 VM on FreeBSD bhyve Layer 1:
Layer 1 VM goes into pause as per original test
<>
2021-05-17 12:31:09.748+: starting up libvirt version: 7.0.0, package:
2ubuntu2 (Christian Ehrhardt Wed, 07 Apr
2021 13:33:46 +0200), qemu version: 5.2.0Debian 1:5.2+dfsg-9ubuntu3, kernel:
5.11.0-17-generic, hostname: green.in.graphica.com.au
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-3-hive-dev-freebsd-12. \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-hive-dev-freebsd-12./.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-hive-dev-freebsd-12./.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-hive-dev-freebsd-12./.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=hive-dev-freebsd-12.2,debug-threads=on \
-S \
-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-hive-dev-freebsd-12./master-key.aes
\
-blockdev
'{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE_4M.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}'
\
-blockdev
'{"driver":"file","filename":"/home/WHO//OVMF_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}'
\
-machine
pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram
\
-cpu
Broadwell-IBRS,vme=on,ss=on,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on
\
-m 4096 \
-object memory-backend-ram,id=pc.ram,size=4294967296 \
-overcommit mem-lock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-uuid 459ff0b9-e0d1-44d4-9862-83315419 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device
pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
\
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-pci-bridge,id=pci.7,bus=pci.1,addr=0x0 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 \
-device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d
\
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-device ide-cd,bus=ide.0,id=sata0-0-0,bootindex=1 \
-blockdev
'{"driver":"file","filename":"/home/WHO//VM-HD.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}'
\
-device ide-hd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1,bootindex=2 \
[Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Thanks for the test, that ensures it still is in 5.2 Unfortunately since a few days that isn't the very most recent version [1] as 6.0 release two weeks ago. I don't have a 6.0 version ready as Ubuntu package yet that I could ask you to try. Usually for an upstream bug report (which IMHO is the right next step) you'd want to have confirmed that the last release is affected as well. So the question IMHO should now be - how do we get you a qemu 6.0 to try. And if confirmed there the next step would be getting in touch with upstream at https://gitlab.com/qemu-project/qemu/-/issues How comfortable (or not) would you feel building your own qemu for a test? It should be something like: $ git clone git://git.qemu.org/qemu.git $ sudo vim /etc/apt/sources.list # edit sources.list to have "# deb-src" lines no more commented out $ sudo apt update $ sudo apt build-dep qemu $ cd qemu $ mkdir build $ cd build # you should need almost nothing for your test, so the following (or similar) should give you a quick build $ ../configure --disable-werror --disable-user --disable-linux-user --disable-docs --disable-guest-agent --disable-sdl --disable-gtk --disable-vnc --disable-xen --disable-brlapi --disable-fdt --disable-bluez --disable-hax --disable-vde --disable-netmap --disable-rbd --disable-libiscsi --disable-libnfs --disable-smartcard --disable-libusb --disable-usb-redir --disable-seccomp --disable-glusterfs --disable-tpm --disable-numa --disable-opengl --disable-virglrenderer --disable-xfsctl --disable-vxhs --disable-slirp --disable-blobs --target-list=x86_64-softmmu --disable-rdma --disable-pvrdma --disable-attr --disable-vhost-net --disable-vhost-vsock --disable-vhost-scsi --disable-vhost-crypto --disable-vhost-user --disable-spice --disable-qom-cast-debug --disable-vxhs --disable-bochs --disable-cloop --disable-dmg --disable-qcow1 --disable-vdi --disable-vvfat --disable-qed --disable-parallels --disable-sheepdog --disable-avx2 --disable-nettle --disable-gnutls --disable-capstone --enable-tools $ make $ sudo make install The above is untested writeup from memory (except the configure line, but that was for a different version) so expect some slight modifications to be needed. You can then replace the qemu in your system (back it up) at /usr/bin/qemu-system-x86_64 with that new built version for a try. I'm currently rather busy, so the delay until I can provide a 6.0 might be a bit. But if you are unable to build your own you can surely wait for that to be ready. Or - also an alternative - you can report it upstream despite not having tested in on 6.0 yet. They might ask for it then, but chances are that someone more familiar with acpi or bhyve immediately recognizes it and can help. If you happen to do so please leave me a link to the issue here so I'm able to track it. [1]: https://wiki.qemu.org/Planning/6.0 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1876678 Title: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1876678/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Hi Christian,
I have re-tested with Ubuntu 21.04 (Hirsute Hippo).
It took me a while to set up test environment.
Summary:
Ubuntu Version:
$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=21.04
DISTRIB_CODENAME=hirsute
DISTRIB_DESCRIPTION="Ubuntu 21.04"
NAME="Ubuntu"
VERSION="21.04 (Hirsute Hippo)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 21.04"
VERSION_ID="21.04"
HOME_URL="https://www.ubuntu.com/;
SUPPORT_URL="https://help.ubuntu.com/;
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/;
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy;
VERSION_CODENAME=hirsute
UBUNTU_CODENAME=hirsute
Linux Version:
$ uname -a
Linux green 5.11.0-17-generic #18-Ubuntu SMP Thu May 6 20:10:11 UTC 2021 x86_64
x86_64 x86_64 GNU/Linux
QEMU / Libvirt Version:
$ sudo virsh version
Compiled against library: libvirt 7.0.0
Using library: libvirt 7.0.0
Using API: QEMU 7.0.0
Running hypervisor: QEMU 5.2.0
Nesting Scenario:
Layer 0 - Ubuntu 21.04
Layer 1 - FreeBSD 12.2 Bhyve Host
Layer 2 - FreeBSD 12.2 Guest
Result:
Virtual Machine Freezes (without work around of turning off APIC
interrupt delivery as per existing diagnosis:
Workaround is to disable APIC virtual interrupt delivery:
1. Add entry into Layer 1 - FreeBSD Guest / Host: /boot/loader.conf:
hw.vmm.vmx.use_apic_vid=0
Here is the libvirt log taken from Layer 0 - Ubuntu host:
<>
2021-05-16 09:57:28.970+: starting up libvirt version: 7.0.0, package:
2ubuntu2 (Christian Ehrhardt Wed, 07 Apr
2021 13:33:46 +0200), qemu version: 5.2.0Debian 1:5.2+dfsg-9ubuntu3, kernel:
5.11.0-17-generic, hostname: green.in.graphica.com.au
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-10-hive-dev-freebsd-12. \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-10-hive-dev-freebsd-12./.local/share
\
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-10-hive-dev-freebsd-12./.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-10-hive-dev-freebsd-12./.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=hive-dev-freebsd-12.2,debug-threads=on \
-S \
-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-10-hive-dev-freebsd-12./master-key.aes
\
-blockdev
'{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE_4M.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}'
\
-blockdev
'{"driver":"file","filename":"/home/jbh/Documents/virtual-machines/hive.dev.freebsd/OVMF_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}'
\
-machine
pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram
\
-cpu
Broadwell-IBRS,vme=on,ss=on,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on
\
-m 4096 \
-object memory-backend-ram,id=pc.ram,size=4294967296 \
-overcommit mem-lock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-uuid 459ff0b9-e0d1-44d4-9862-83315419 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device
pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
\
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-pci-bridge,id=pci.7,bus=pci.1,addr=0x0 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 \
-device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d
\
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-device ide-cd,bus=ide.0,id=sata0-0-0,bootindex=1 \
-blockdev
'{"driver":"file","filename":"/home/jbh/Documents/virtual-machines/hive.dev.freebsd/hive-hd1-01.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}'
\
-blockdev
[Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Hi Christian, just letting you know I have got email notifications and will re-run tests. It will likely take me a couple days to complete this. I will post findings once done. I will try against 20.04 and 21.04 to start and post on various component versions and results. Cheers, John. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1876678 Title: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1876678/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Hi John, could you give it a try with the more recent virtualization stack in [1]. Since this might as well be in the kernel and not qemu/libvirt you might also consider checking other kernel versions - not sure with your self-built driver, but what kernels have you tried and which newer ones could you try? If you can overcome the other issue in another way you might try [2] which is great to check various versions. That works "in place" on your 20.04 system and if better would indicate that one of the components has a fix that we only need to identify. P.S. the PPA does not yet contain qmeu 6.0 which released a few days ago, it will be june until I get to that I guess :-/ [1]: https://launchpad.net/~canonical-server/+archive/ubuntu/server-backports [2]: https://kernel.ubuntu.com/~kernel-ppa/mainline/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1876678 Title: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1876678/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Since you were talking about Ubuntu, I moved this to the Ubuntu tracker now. If you can reproduce the problem with upstream QEMU (currently v6.0), then please open a new ticket in the new QEMU issue tracker at gitlab.com. ** Project changed: qemu => qemu (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1876678 Title: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1876678/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
