Launchpad has imported 8 comments from the remote bug at https://bugzilla.redhat.com/show_bug.cgi?id=1380803.
If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. ------------------------------------------------------------------------ On 2016-09-30T15:03:37+00:00 Alan wrote: Description of problem: Running "ip link show" outputs Message Truncated several times and the resulting interface list is incomplete. The issue occurs when the number of VF interfaces is over ~50. This is impacting applications/scripts which rely on parsing the output of ip to ascertain the current network configuration (e.g. ovirt/vdsm) Version-Release number of selected component (if applicable): iproute-3.10.0-54.el7_2.1.x86_64 This appears to be related to https://bugzilla.redhat.com/show_bug.cgi?id=1086512 Reply at: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/comments/0 ------------------------------------------------------------------------ On 2016-10-26T10:59:36+00:00 Alan wrote: This has been fixed upstream in either 4.5 or 4.6. Rebuilding the SRPM from Fedora 25 (iproute-4.6.0-1.fc25.src.rpm) fixes the problem. But iproute-4.4.0-3.fc24.src.rpm from Fedora 24 does not. Reply at: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/comments/1 ------------------------------------------------------------------------ On 2017-01-11T02:42:06+00:00 Hangbin wrote: commit 72b365e8e0fd5efe1d5c05d04c25950736635cfb Author: Phil Sutter <p...@nwl.cc> Date: Fri Mar 4 19:57:28 2016 +0100 libnetlink: Double the dump buffer size There have been reports about 'ip addr' printing "Message truncated" on systems with large numbers of VFs. Although I haven't been able to get my hands on hardware suitable to reproduce this, increasing the dump buffer has been reported to resolve the issue. For want of a better idea, just double the buffer size to 32k. Feels like this opportunistic buffer size selection is rather workarounding a design flaw in libnetlink or maybe even the netlink protocol itself. Signed-off-by: Phil Sutter <p...@nwl.cc> Reply at: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/comments/2 ------------------------------------------------------------------------ On 2017-03-24T10:32:00+00:00 Alan wrote: If there's any interest I could probably provide access to a system exhibiting the issue (Cisco B200 with VIC1340). Reply at: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/comments/3 ------------------------------------------------------------------------ On 2017-07-05T15:37:23+00:00 Alan wrote: It seems that increasing the buffer size has simply pushed the problem further up. With ~95 VFs the message truncated warnings return. On my Cisco UCS platform it's possible to go as high as 114 VFs, while other configurations support up to 223. Reply at: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/comments/4 ------------------------------------------------------------------------ On 2017-07-21T12:23:31+00:00 Phil wrote: Hi Alan, (In reply to Alan Griffiths from comment #5) > It seems that increasing the buffer size has simply pushed the problem > further up. With ~95 VFs the message truncated warnings return. > > On my Cisco UCS platform it's possible to go as high as 114 VFs, while other > configurations support up to 223. So you see the problem even with iproute-4.6.0-1.fc25? Thanks, Phil Reply at: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/comments/5 ------------------------------------------------------------------------ On 2017-07-25T11:17:34+00:00 Alan wrote: Yes, so the behaviour seems to be the same across the releases now. I've tested: - iproute-3.10.0-74 (EL7) iproute-4.6.0-1 (FC25) iproute-4.11.0-1 (FC26) The test system has 2 PF. At 90 x VF all above versions work, at 91 x VF I start seeing message truncated errors. Reply at: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/comments/6 ------------------------------------------------------------------------ On 2017-10-10T11:53:00+00:00 Jan wrote: Just a note: There are theoretical and practical upper limits to the number of Virtual Functions per PF (256 from some sources)[1], so setting a static maximum buffer size is a justifiable option. That moves the burden over to ensuring that the response per VF doesn't grow in the future. Libvirt had similar issues (it used libnl) until it enabled message peeking by default. Unfortunately the only way to solve this issue in all cases without performance penalty is to alter the kernel's message truncation handling mechanism to not free the skb if MSG_TRUNC occurred. [1] It's possible to put in a retry on truncation, resizing the receive buffer to fit the response, BUT since that's two unrelated syscalls, it leaves me feeling a bit queasy. [1] http://windowsitpro.com/systems-management/q-what-sr-iov [2] http://man7.org/linux/man-pages/man2/recv.2.html Reply at: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/comments/13 ** Changed in: iproute2 (CentOS) Status: Unknown => In Progress ** Changed in: iproute2 (CentOS) Importance: Unknown => High ** Bug watch added: Red Hat Bugzilla #1086512 https://bugzilla.redhat.com/show_bug.cgi?id=1086512 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to iproute2 in Ubuntu. https://bugs.launchpad.net/bugs/1720126 Title: [ip link] Message truncated error for large number of passthrough VFs Status in iproute2 package in Ubuntu: Fix Released Status in iproute2 source package in Trusty: Fix Committed Status in iproute2 source package in Xenial: Fix Committed Status in iproute2 source package in Zesty: Fix Released Status in iproute2 package in CentOS: In Progress Bug description: [Impact] When querying a Physical Function netdev with a large amount of VF's (more than 30), the resulting return message can overflow the 16K netlink message buffer. This can be fixed by enabling message peeking on the socket and resizing the buffer on receive, or by simply enlarging the receive buffer. Since there's an upper limit to the number of VF's per PF, it's relatively sane to just enlarge the receive buffer. Please see the attached patch. [Test Case] # Set up 60 VF's on an SR-IOV device ip link show > /dev/null Observe the following: Message truncated Message truncated Message truncated [Regression Potential] 1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case. 2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny). 3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load. [Other Info] Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04 ===================================================================================================== Ubuntu16 system stack@cluster04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial stack@cluster04:~$ uname -r 4.4.0-93-generic stack@cluster04:~$ apt-cache policy iproute2 iproute2: Installed: 4.3.0-1ubuntu3.16.04.1 Version table: *** 4.3.0-1ubuntu3.16.04.1 500 500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages ================================================================================================= Ubuntu14 system: root@boomslang:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty root@boomslang:~# uname -r 4.4.0-96-generic root@boomslang:~# apt-cache policy iproute2 iproute2: Installed: 3.12.0-2ubuntu1 Version table: *** 3.12.0-2ubuntu1 0 500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
