[Bug 1978489] Re: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1978489 Title: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1978489/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1978489] Re: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
This bug was fixed in the package nova - 3:25.2.1-0ubuntu2~cloud0 --- nova (3:25.2.1-0ubuntu2~cloud0) focal-yoga; urgency=medium . * New update for the Ubuntu Cloud Archive. . nova (3:25.2.1-0ubuntu2) jammy; urgency=medium . * d/p/libvirt-remove-default-cputune-shares-value.patch: Enable launch of instances with more than 9 CPUs on Jammy (LP: #1978489). ** Changed in: cloud-archive/yoga Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1978489 Title: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1978489/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1978489] Re: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
This bug was fixed in the package nova - 3:25.2.1-0ubuntu2 --- nova (3:25.2.1-0ubuntu2) jammy; urgency=medium * d/p/libvirt-remove-default-cputune-shares-value.patch: Enable launch of instances with more than 9 CPUs on Jammy (LP: #1978489). -- Corey Bryant Tue, 16 Jan 2024 12:30:33 -0500 ** Changed in: nova (Ubuntu Jammy) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1978489 Title: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1978489/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1978489] Re: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
comment #17 stated that noble and mantic have the patch, so I'm marking the noble (devel) task as fix released. ** Changed in: nova (Ubuntu) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1978489 Title: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1978489/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1978489] Re: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
Forget to add to ^ that instead of removing the default weight (1024 * guest.vcpus) might it not have made sense to simply cap it at the max allowed value? Again, perhaps something that could be proposed to Nova as a new patch. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1978489 Title: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1978489/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1978489] Re: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
As a recap, this patch addresses the problem of moving vms between hosts running cgroups v1 (e.g. Ubuntu Focal) and v2 (Ubuntu Jammy) which now has a cap of 10K [1] for cpu.weight, resulting in vms with > 9 vcpus not being able to boot if they use the default Nova 1024 * guest.vcpus. The patch addresses the problem by no longer applying a default weight to instances while keeping the option to apply quota:cpu_shares from a flavor extra-specs. The consequence of this is: Vms booted without quota:cpu_shares extra-specs after upgrading to this patch will have the default cgroups v2 weight of 100. New Vms can get a higher weight if they use a flavor with extra-specs quota:cpu_shares BUT this will only apply to existing vms if they are resized so as to switch to using the new/modified flavor which will require workload downtime - a vm reboot will not consume the new value. Vms created from a flavor with extra-specs quota:cpu_shares set to a value > 10K will fail to boot and to fix this will require a new/modified flavor with adjusted value then vm resize to consume therefore workload downtime. It is important to note that point 3 is not a consequence of this patch and is therefore neither introduced nor resolved by it and will require a separate patch solution. One way to resolve this could be to have Nova cap quota:cpu_shares at cgroup cpu.weight max value and log a warning to say that was done, that way instances will at least boot and have a max weight. Therefore I am in favour of proceeding with this SRU to provide users a way to migrate from v1 to v2 and suggest we propose a new patch to address the flavor extra-specs issue. As @jamespage has pointed out there are some interim manual solutions that can be used as a stop-gap until this is fully resolved in Nova. [1] https://www.kernel.org/doc/Documentation/cgroup-v2.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1978489 Title: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1978489/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1978489] Re: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
Re: > The same patch should also be available on cloud archive cloud:focal- yoga This will happen alongside the changes being made into 22.04 - the updates are in the yoga-proposed pocket at the moment. ** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/yoga Importance: Undecided Status: New ** Changed in: cloud-archive Status: New => Invalid ** Changed in: cloud-archive/yoga Status: New => Fix Committed ** Changed in: cloud-archive/yoga Importance: Undecided => High ** Changed in: nova (Ubuntu Jammy) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1978489 Title: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1978489/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1978489] Re: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
I think that the challenge of how to update the cpu tuning for all existing running instances is solvable. a) quota:cpu_* is an additional property for a flavor and as such can be updated (applying to new instances created). b) Using the virsh tool, its possible to live set the scheduling tuning on a running instance - for example: sudo virsh schedinfo instance-0008905c --config --live --set cpu_shares=2048 That obviously needs tailoring for the actual running environment/instances. That does not however deal with the in-balance between instances created before and post update with no flavor extra-specs defined. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1978489 Title: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1978489/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1978489] Re: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
> This behavior can be recovered by setting the quota:cpu_shares flavor extra spec. You are the openstack experts here, but I will point out that it looks like comment #10 already tried this. That comment also ends with: "Is there any workaround to rebuilding hundreds of instances like force nova to override flavors of existing instances?" Do we have a concrete answer for that? Like, "here is what you do, step by step". -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1978489 Title: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1978489/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
