[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
I think I have encountered exactly the same issue as you Nico. We have very similar setups. Upgrading the kernel did not help. Only thing that helped was setting openstack to use qemu instead of kvm in L2 VMs with the performance cost associated with doing that :( -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797332 Title: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
Thank you for your suggestion @Alexandru ! . . . I can not try this fix because since then I have moved on and I use Ubuntu18.04 for my L0 hypervisor, and I have also tried with Ubuntu18.04 on th L1 VMs. . . . However very interesting. On my previous Ubuntu16.04 hosts, I believe I used "linux-image-4.4.0-XYZ-generic". -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797332 Title: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
FWIW, bumping the kernel on the host (and most likely on the L1 VMs too) should work. The HWE kernel in Xenial is the same version (4.15) with the kernel used by Bionic (18.04), so this should fix the problem: $ apt install linux-generic-hwe-16.04 $ reboot BR, Alex -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797332 Title: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
[Expired for qemu (Ubuntu) because there has been no activity for 60 days.] ** Changed in: qemu (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797332 Title: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
You say:
> What version of libvirt is running in your lvl1 (or all levels)?
cf my 1st post and below:
On my Ubuntu16.04 bare metal host:
```
libvirt-bin 4.0.0-1ubuntu8.5~cloud0
qemu-system-x86 1:2.11+dfsg-1ubuntu7.6~cloud0
```
On one Ubuntu16.04 lvl1 VM:
```
libvirt-bin 4.0.0-1ubuntu8.3~cloud0
qemu-system-x86 1:2.11+dfsg-1ubuntu7.4~cloud0
```
.
.
.
.
You say:
> - [can you] reproduce the same on a different host CPU
If I can I will try, on a much 'smaller' device (not a Xeon CPU).
Maybe tomorow.
.
.
.
.
You say:
> To confirm our current theory that the usage/emulation/nesting of invpcid
is the root cause [...]
Dumpxml of my lvl2 VM with "":
https://paste.ubuntu.com/p/WxvfBcHnF2/
And here is the boot log of the lvl2 VM with invpcid disabled:
The VM failed to boot. Maybe I missed sth.
https://paste.ubuntu.com/p/bkqDsT8VTy/
I followed [1]:
[1] https://youth2009.org/post/kvm-with-ubuntu-cloud-image/
.
.
.
.
you say:
> NB-reply: there was no --os-variant 18.04 released back then
No big deal, it is just frustrating to instantiate an Ubuntu18.04 and to set
this parameter to something else !
.
.
.
.
You say:
> x86 nested virt is never really supported, just "as good as it happens to
work". I wonder if that is one of those cases.
First I thought nested virt could make my life easier regarding what I wanted
to achieve. There are several ways of testing and deploying OpenStack.
With the way I chose, I could 'simulate' a multi-hosts environment like in [2]
with several compute nodes, etc...
[2] https://github.com/nuagenetworks/nuage-openstack-ansible/wiki
/Configure-OSA-Multi-node-Environment
But now I understand that nested virt is maybe too much a beta.
I will try with "Ubuntu 18.04 > Ubuntu 16/18 > {whatever OS}".
Maybe this is patched in Ubuntu18.04.
And if it does not suit my needs, I will figure out something else (and more
bare metal ^^).
Metal as a Service (Ubuntu MAAS) looks good, but it is too much for me.
.
.
.
.
TY for your help. I think this bug is hard to identify and maybe harder to
patch. And I am not a virtualization or qemu or OpenStack expert
(for the moment !?). So I can't help you more.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1797332
Title:
qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
Ok, at lvl1 definition Openstack came up with it's cpu modelling which in this case is actually: cpu mode='host-passthrough + a bunch of required features That is what gives your LVL1 the invpcid feature (so far so good). At lvl2 we have Nova: vs uvtool Thanks for the data Nicolas! With that in mind I have set my LVL1 to run the same host-passthrough config that you have reported. Then again I configure LVL2 to run the same host-model config. Note: "my 16.04" would not allow "check='partial'", so I dropped it. What version of libvirt is running in your lvl1 (or all levels)? Current is 1.3.1-1ubuntu10.24 I was feeling glad that it seems that the uvtool style guests work for you as I assumed. But even with the same CPU definitions used in my case it works for me. That is for "16.04 > 16.04 > 16.04" as well. x86 nested virt is never really supported, just "as good as it happens to work". I wonder if that is one of those cases. My chip is a somewhat older 12 core "Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz". We have plenty of SW workarounds already, but if you can spend the time I wonder if you can: - reproduce the same on a different host CPU - To confirm our current theory that the usage/emulation/nesting of invpcid is the root cause, could you on the failing case in the definition for LVL2 add to the cpu section. That would keep the rest as-is, but remove that feature. NB-reply: there was no --os-variant 18.04 released back then, but since there was no change since former releases it doesn't matter - the only drawback is that sometimes people are wondering if it is missing. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797332 Title: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
On my previous comment, I am in the "16.04 > 16.04 > 16.04" situation. I say this especially for my 3 dumpxml files. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797332 Title: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
Hi Christian, First, I tried to create a lvl2 VM using your suggestion with `uvtool-libvirt`. I have tried this on one of my lvl1 VM, which is an OpenStack compute node. ``` compute@L1: $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=xenial label=daily compute@L1: $ uvt-kvm create --memory 4096 --disk 30 --cpu 4 --password ubuntu xenial-guest-lvl1 arch=amd64 release=xenial label=daily ``` And this way, it works ! However, if I use the OpenStack API to create a lvl2 VM on this same compute node, the OpenStack Nova VM fails. . . . . If we recap the 3 options tested here to create an lvl2 VM: 1. OpenStack API -> FAIL The lvl2 VM is stuck for example at: "Starting Update UTMP about System Boot/Shutdown..." 2. virt-install "By hand" -> FAIL If I do like in [1], VM generate the same error than in my 1st post. 3. uvtool-libvirt -> SUCCESS Your example works just fine. [1] https://youth2009.org/post/kvm-with-ubuntu-cloud-image/ . . . . You say: > That guest runs just fine and is happy. > So it has to be part of your guest configuration in some way. I agree: maybe I should look further into the options of `virt-install`. What I give in my first post (the virt-install script) is my way of creating lvl1 VM. NB: It seems that for the moment, --os-variant has no `ubuntu18.04` value. I keep this parameter to ubuntu16.04, even if I want to create a 18.04 VM. The difference between Ubuntu 16.04 and 18.04 regarding `virt-install`: * 16.04: virt-install --version is 1.3.2 * 18.04: virt-install --version is 1.5.1 So maybe the problem comes from `virt-install` and the way I configure a VM. . . . . However when looking at the OpenStack API, here I am not the one who provides the guest configuration. I provide the OpenStack API with the info it needs to create a new OpenStack instance (i.e. flavor, image type, cloud-init config, etc...) and then the API ~converts~ this description to instantiate this OpenStack instance on the compute node which is running qemu/KVM. I am not sure what the OpenStack API uses to do that. I assume it uses python-libvirt [2] but I may be wrong. [2] https://libvirt.org/docs/libvirt-appdev-guide-python/en-US/html/libvirt_application_development_guide_using_python-Guest_Domains-Lifecycle_Control.html#libvirt_application_development_guide_using_python-Guest_Domains-Lifecycle-Provisioning_and_Starting . . . . You say: > I wonder if your way to setup the guests uses special CPU types [...] > > Waiting for your feedback on guest CPU definitions in your case. My CPU is a Intel Xeon Broadwell. On the lvl0, which have 48 cores: ``` baremetal@L0:cat /proc/cpuinfo | grep invpcid | wc -l 48 ``` On the lvl1, which is a VM with 20 vCPU: ``` compute@L1:$ cat /proc/cpuinfo | grep invpcid | wc -l 20 ``` . . . . Dumpxml of a working lvl1 VM "compute41": https://paste.ubuntu.com/p/KMrCKGgvRg/ Dumpxml of a failing lvl2 VM created by the OpenStack/Nova API on "compute41": https://paste.ubuntu.com/p/9FrhMWWgVk/ Dumpxml of a working lvl2 VM created by uvtool-libvirt: https://paste.ubuntu.com/p/4CztPDW7fM/ On difference I see with your Dumpxml is for os part: machine='pc-i440fx-bionic' for me machine='pc-i440fx-xenial' for you Maybe this is due to the way I install qemu with cloud-archive:queens [3]. [3] https://wiki.ubuntu.com/OpenStack/CloudArchive . . . . qemu logs for the lvl2 VM created by uvtool-libvirt: ``` compute@L1:$ cat /var/log/libvirt/qemu/xenial-guest-lvl2.log [...] 2018-10-12T14:28:03.317760Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.8001H:ECX.svm [bit 2] ``` If I miss sth, let me know! -- Nicolas -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797332 Title: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
Hi Nicolas, interesting. Seeing CPUID.07H:EBX.invpcid makes me wonder - IIRC that was a speedup feature long neglected by everyone but suddenly becoming important in the context of meltdown avoidance. Maybe that wasn't passed/emulated in the older qemu but the guest now insists or misdetects it? This also would sort of match your statement "It worked some times ago, but not anymore." as the meltdown fixes obviously came after the release of 16.04. Let me try to recreate your initial case first with 16.04->16.04->16.04. I took a fresh deployed Xenial host and deployed a Xenial guest in lvl1 $ sudo apt install uvtool-libvirt $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=xenial label=daily $ uvt-kvm create --memory 4096 --disk 30 --cpu 4 --password ubuntu xenial-guest-lvl1 arch=amd64 release=xenial label=daily And then in the lvl1 guest doing the same to spawn a smaller lvl2 guest. ... # note: back then (16.04) nested default libvirt network needed to manually get to work before the next command $ uvt-kvm create --password ubuntu xenial-guest-lvl2 arch=amd64 release=xenial label=daily That guest runs just fine and is happy. So it has to be part of your guest configuration in some way. $ cat /proc/cpuinfo | grep invpcid Report it is available on the Host (lvl0) but none of the guests (lvl1/lvl2). I did not see a warning like yours about CPUID.07H:EBX.invpcid (on neither of the lvls). My guests are defined the "most default" way possible leaving most of the cpu construction to the defaults of libvirt/qemu. virsh dumpxml content: lvl1 => http://paste.ubuntu.com/p/fH57d5prmS/ lvl2 => http://paste.ubuntu.com/p/vQbcgfmfVv/ I wonder if your way to setup the guests uses special CPU types that define the meltdowny features - like the -IBRS types or even adding features like those mentioned in [1]. E.g. Virt-manager would default to "Haswell-noTSX-IBRS" on my system with the virt stack of 16.04. If I use that in my guest definition (on both levels) Haswell-noTSX-IBRS Now I get invpcid in $ cat /proc/cpuinfo | grep invpcid in the lvl1 guest. But since this type lacks the KVM features I'm no more assuming but waiting for your reply on how guest CPU is modelled in your case. But in general in that case i could think of this being a potential trouble for (x86) nesting which is generally known as "working great until it does't" Waiting for your feedback on guest CPU definitions in your case. [1]: https://www.berrange.com/posts/2018/06/29/cpu-model-configuration- for-qemu-kvm-on-x86-hosts/ ** Changed in: qemu (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797332 Title: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
[update] I tested some new combinations #1 and #2 (see in attachment) with Ubuntu 18.04 and 16.04. I think for the moment and if it fits my needs, I will stick to combination #1 and/or #2. ** Attachment added: "nestedRecap.md" https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+attachment/5200028/+files/nestedRecap.md -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797332 Title: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU
** Summary changed: - qemu nested virtualization is not working with Ubuntu16.04 + qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797332 Title: qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
