[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
** Tags removed: sts-sru -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
** No longer affects: libnl3 (Ubuntu Precise) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
This bug was fixed in the package libnl3 - 3.2.21-1ubuntu4 --- libnl3 (3.2.21-1ubuntu4) trusty; urgency=high [ Jorge Niedbalski ] * d/p/lib-nl-Increase-receive-buffer-size-to-4-pages.patch: Increase receive buffer size to 4 pages by default. (LP: #1567578). -- Louis Bouchard Mon, 24 Oct 2016 16:26:30 +0200 ** Changed in: libnl3 (Ubuntu Trusty) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
Hello Bryan, After installing the -proposed version the error presented on this bug is not reproducible anymore and the guest starts correctly. You can follow the reproduction steps that I added to the bug's description. Without -proposed: $ sudo virsh destroy guest1 Domain guest1 destroyed $ sudo virsh start guest1 error: Failed to start domain guest1 error: internal error: missing IFLA_VF_INFO in netlink response With proposed: root@bronzor:/home/ubuntu# start libvirt-bin libvirt-bin start/running, process 31492 root@bronzor:/home/ubuntu# virsh start guest1 Domain guest1 started ** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
** Description changed: - [Description] + [Impact] - Ubuntu 14.04.4 and SRIOV settings. + libnl can only enable up to 30 VFs even if the PF supports up to 63 VFs + in an Openstack SRIOV configuration. As already documented in https://bugs.launchpad.net/mos/+bug/1501738 there is a bug in the default libnl library release installed on Ubuntu - 14.04.4 + 14.04.4. When trying to enable a guest with more than 30 VFs attached, the following error is returned: error: Failed to start domain guest1 error: internal error: missing IFLA_VF_INFO in netlink response - [Impact] - - The library release is the 3.2.21-1 the bug is impacting on the maximum - VFs number that can be enabled (up to 30) even if the PF supports up to - 63 VFs in an Openstack SRIOV configuration - [Test Case] - The sequence to reproduce this bug is: - - 1) Edit /etc/default/grub + 1) Edit /etc/default/grub. GRUB_CMDLINE_LINUX="intel_iommu=on ixgbe.max_vfs=63" - 2) $ sudo update-grub + 2) Update grub and reboot the machine. - ### Reboot the machine. + $ sudo update-grub - 3) Check that the virtual functions are available: + 3) Check that the virtual functions are available. $ sudo lspci|grep -i eth | grep -i virtual | wc -l 126 - 4) Create a KVM guest + 4) Create a KVM guest. $ sudo uvt-kvm create guest1 release=trusty - 5) List the VF devices : + 5) List the VF devices. $ sudo lspci|grep -i eth | grep -i virtual | awk '{print $1}' | sed 's/\:/\_/g' | sed 's/\./\_/g' > devices.txt - 6) Get the libvirt node device: + 6) Get the libvirt node device. $ sudo for device in $(cat ./devices.txt); do virsh nodedev-list | grep $device; done > pci_devices.txt - 7) Generate the XML config for each device: + 7) Generate the XML config for each device. $ sudo mkdir devices && for d in $(cat pci_devices.txt); do virsh nodedev-dumpxml $d > devices/$d.xml; done - 8) Save and Run the following script + 8) Save and Run the following script. (http://pastebin.ubuntu.com/23374186/) $ sudo python generate-interfaces.py |grep address | wc -l - 9) Finally attach the devices to the guest. + 9) Finally attach the devices to the guest. $ sudo for i in $(seq 0 63); do virsh attach-device guest1 ./interfaces/$i.xml --config; done Device attached successfully [...] Device attached successfully Device attached successfully - 10) Then destroy/start the guest again, at this point the error is + 10) Then destroy/start the guest again, at this point the error is reproduced. $ sudo virsh destroy guest1 Domain guest1 destroyed $ sudo virsh start guest1 error: Failed to start domain guest1 error: internal error: missing IFLA_VF_INFO in netlink response - * detailed instructions how to reproduce the bug - - * these should allow someone who is not familiar with the affected -package to reproduce the bug and verify that the updated package fixes -the problem. - [Regression Potential] - ** None identified. + * None identified. [Other Info] - - Redhat Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1040626 + * Redhat Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1040626 - [Workaround] + * A workaround is to install a newer library release. - The workaround is to install a newer library release, the 3.2.24-2: - - wget https://launchpad.net/ubuntu/+archive/primary/+files/libnl-3-200_3.2.24-2_amd64.deb - wget https://launchpad.net/ubuntu/+archive/primary/+files/libnl-genl-3-200_3.2.24-2_amd64.deb - wget https://launchpad.net/ubuntu/+archive/primary/+files/libnl-route-3-200_3.2.24-2_amd64.deb - dpkg -i libnl-3-200_3.2.24-2_amd64.deb - dpkg -i libnl-genl-3-200_3.2.24-2_amd64.deb - dpkg -i libnl-route-3-200_3.2.24-2_amd64.deb + $ wget https://launchpad.net/ubuntu/+archive/primary/+files/libnl-3-200_3.2.24-2_amd64.deb + $ wget https://launchpad.net/ubuntu/+archive/primary/+files/libnl-genl-3-200_3.2.24-2_amd64.deb + $ wget https://launchpad.net/ubuntu/+archive/primary/+files/libnl-route-3-200_3.2.24-2_amd64.deb + $ dpkg -i libnl-3-200_3.2.24-2_amd64.deb + $ dpkg -i libnl-genl-3-200_3.2.24-2_amd64.deb + $ dpkg -i libnl-route-3-200_3.2.24-2_amd64.deb ** Changed in: libnl3 (Ubuntu Precise) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
Hello Lorenzo, or anyone else affected, Accepted libnl3 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libnl3/3.2.21-1ubuntu4 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: libnl3 (Ubuntu Trusty) Status: In Progress => Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
** Tags added: sts-sru -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
** Patch added: "fix-1567578-trusty.debdiff" https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+attachment/4766476/+files/fix-1567578-trusty.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
Hello, After running a bisect between tags libnl3_2_21 and libnl3_2_22 I identified the fixer commit to be 807fddc nl: Increase receive buffer size to 4 pages commit 807fddc4cd9ecb12ba64e1b7fa26d86b6c2f19b0 Author: Thomas Graf Date: Wed May 8 13:52:27 2013 +0200 nl: Increase receive buffer size to 4 pages Assuming that the kernel does not send more than a page is no longer valid, and enabling MSG_PEEK'ing by default to figure out the exact message buffer requirements can have a negative influence on the performance of existing applications. Bumping the default receive buffer space to 4 pages seems a sane default. Signed-off-by: Thomas Graf --- After applying this patch on top of the current trusty-updates this problem is not longer exhibited and I can attach the full 128 VFs to the guest. I am proposing this patch for SRU, and I already updated the description with the reproduction steps. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
Hello, I am able to reproduce this bug consistently on a Trusty (3.19.0-73-generic) machine equipped with 2 ixgbe 10GbE cards: 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) 04:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) The sequence to reproduce this bug is: 1) Edit /etc/default/grub GRUB_CMDLINE_LINUX="intel_iommu=on ixgbe.max_vfs=63" 2) $ sudo update-grub ### Reboot the machine. 3) Check that the virtual functions are available: $ sudo lspci|grep -i eth | grep -i virtual | wc -l 126 4) Create a KVM guest $ sudo uvt-kvm create guest1 release=trusty 5) List the VF devices : $ sudo lspci|grep -i eth | grep -i virtual | awk '{print $1}' | sed 's/\:/\_/g' | sed 's/\./\_/g' > devices.txt 6) Get the libvirt node device: $ sudo for device in $(cat ./devices.txt); do virsh nodedev-list | grep $device; done > pci_devices.txt 7) Generate the XML config for each device: $ sudo mkdir devices && for d in $(cat pci_devices.txt); do virsh nodedev-dumpxml $d > devices/$d.xml; done 8) Save and Run the following script (http://pastebin.ubuntu.com/23374186/) $ sudo python generate-interfaces.py |grep address | wc -l 9) Finally attach the devices to the guest. $ sudo for i in $(seq 0 63); do virsh attach-device guest1 ./interfaces/$i.xml --config; done Device attached successfully [...] Device attached successfully Device attached successfully 10) Then destroy/start the guest again, at this point the error is reproduced. $ sudo virsh destroy guest1 Domain guest1 destroyed $ sudo virsh start guest1 error: Failed to start domain guest1 error: internal error: missing IFLA_VF_INFO in netlink response Please note that this error is not reproducible using libnl3_2_22. I am currently working to bisect and identify the offending commit and propose a backport to Ubuntu Trusty. ** Description changed: + [Description] + Ubuntu 14.04.4 and SRIOV settings. As already documented in https://bugs.launchpad.net/mos/+bug/1501738 there is a bug in the default libnl library release installed on Ubuntu 14.04.4 - The library release is the 3.2.21-1 and the bug is impacting on the - maximum VFs number that can be enabled (up to 30) even if the PF - supports up to 63 VFs in an Openstack SRIOV configuration + When trying to enable a guest with more than 30 VFs attached, the + following error is returned: + + error: Failed to start domain guest1 + error: internal error: missing IFLA_VF_INFO in netlink response + + [Impact] + + The library release is the 3.2.21-1 the bug is impacting on the maximum + VFs number that can be enabled (up to 30) even if the PF supports up to + 63 VFs in an Openstack SRIOV configuration + + [Test Case] + + The sequence to reproduce this bug is: + + 1) Edit /etc/default/grub + + GRUB_CMDLINE_LINUX="intel_iommu=on ixgbe.max_vfs=63" + + 2) $ sudo update-grub + + ### Reboot the machine. + + 3) Check that the virtual functions are available: + + $ sudo lspci|grep -i eth | grep -i virtual | wc -l + 126 + + 4) Create a KVM guest + + $ sudo uvt-kvm create guest1 release=trusty + + 5) List the VF devices : + + $ sudo lspci|grep -i eth | grep -i virtual | awk '{print $1}' | sed + 's/\:/\_/g' | sed 's/\./\_/g' > devices.txt + + 6) Get the libvirt node device: + + $ sudo for device in $(cat ./devices.txt); do virsh nodedev-list | grep + $device; done > pci_devices.txt + + 7) Generate the XML config for each device: + + $ sudo mkdir devices && for d in $(cat pci_devices.txt); do virsh + nodedev-dumpxml $d > devices/$d.xml; done + + 8) Save and Run the following script + (http://pastebin.ubuntu.com/23374186/) + + $ sudo python generate-interfaces.py |grep address | wc -l + + 9) Finally attach the devices to the guest. + + $ sudo for i in $(seq 0 63); do virsh attach-device guest1 ./interfaces/$i.xml --config; done + Device attached successfully + [...] + + Device attached successfully + Device attached successfully + + 10) Then destroy/start the guest again, at this point the error is + reproduced. + + $ sudo virsh destroy guest1 + Domain guest1 destroyed + + $ sudo virsh start guest1 + + error: Failed to start domain guest1 + error: internal error: missing IFLA_VF_INFO in netlink response + + * detailed instructions how to reproduce the bug + + * these should allow someone who is not familiar with the affected +package to reproduce the bug and verify that the updated package fixes +the problem. + + [Regression Potential] + + ** None identified. + + [Other Info] + + - Redhat Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1040626 + + [Workaround] The workaround is to install a newer library release, the 3.2.24-2: wget https://launchpad.net/ubuntu/+archive/primary/+files/libnl-3-200_3.2.24-2_amd64.deb wget https://launchpad.net/ubuntu/+archive/primary/+files/libnl-genl-3-200_3.2.24-2_amd64.deb wget https://launchpad.net/u
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
** Changed in: libnl3 (Ubuntu Trusty) Status: New => In Progress ** Changed in: libnl3 (Ubuntu Trusty) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
** Changed in: libnl3 (Ubuntu) Status: Confirmed => Fix Released ** Changed in: libnl3 (Ubuntu) Assignee: Jorge Niedbalski (niedbalski) => (unassigned) ** Changed in: libnl3 (Ubuntu Trusty) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: libnl3 (Ubuntu Precise) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
** Also affects: libnl3 (Ubuntu Precise) Importance: Undecided Status: New ** Also affects: libnl3 (Ubuntu Trusty) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
** Changed in: libnl3 (Ubuntu) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: libnl3 (Ubuntu) Importance: Low => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
The biggest issue the libnl upgrade will fix are the following libvirt errors with 3.13.0 LTS kernel: internal error: missing IFLA_VF_INFO in netlink response Hence the package should be updated. I had those issues with the xenial LTS kernel too until the libnl was updated -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
** Tags removed: 14.04.4 libnl ubuntu vf ** Tags added: trusty ** Changed in: libnl3 (Ubuntu) Importance: Undecided => Low -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
I was not able to reproduce the problem using trusty + 3.19 kernel and libvirt alone. I was able to scale appropriately to 63 domains using VFs. I guess next step is to test the OpenStack instructions identified, but then it may be a problem lying elsewhere. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1567578] Re: libnl should be updated to support up to 63 VFs per single PF
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: libnl3 (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567578 Title: libnl should be updated to support up to 63 VFs per single PF To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1567578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs