Public bug reported:
Description
------------
It can be read in SR-IOV companion guide that 82599 chipsets :
* The 82599 has 128 Transmit and 128 Receive queues. They are generally
referred to/thought of as queue pairs (1 Transmit and 1 Receive queue). This
gives the 82599 128 queue pairs.
* The 82599 has up to 64 pools, with each pool having two queue pairs in
it. That is two Transmit and Receive queue assigned to each VF.
* The 82599 allows for a variable configuration for the number of pools. In
SR-IOV mode, there can be 16, 32 or 64 pools.
The last line suggest using combined queues of 2, 4 and 8.
The only way, for now we found, is using tc on the host especially as the
ethtool command is not supported (ethtool -L eth10 combined 8 for example).
And, with following command "tc qdisc add dev eth10 root mqprio num_tc 8", we
will be able to use up to 8 combined queues on the host PF and guests which are
using VFs.
Yet, with 8 combined queues with tc commnad, VLAN creation was not permitted.
Following driver versions have been used on host and guest to enable this
multiqueue option:
* 4.3.15 ixgbe in the host
* 3.1.2 ixgbevf in the guest
To reproduce
-------------
On the host side
----------------
root@ubuntu1404:~# rmmod ixgbe
root@ubuntu1404:~# modprobe ixgbe
[ 4817.027637] ixgbe 0000:05:00.0: failed to load because an unsupported SFP+
or QSFP module type was detected.
[ 4817.038630] ixgbe 0000:05:00.0: Reload the driver after installing a
supported module.
[ 4817.418360] ixgbe 0000:41:00.1: failed to load because an unsupported SFP+
or QSFP module type was detected.
[ 4817.429354] ixgbe 0000:41:00.1: Reload the driver after installing a
supported module.
[ 4817.838086] ixgbe 0000:43:00.0: failed to load because an unsupported SFP+
or QSFP module type was detected.
[ 4817.849091] ixgbe 0000:43:00.0: Reload the driver after installing a
supported module.
[ 4817.889189] ixgbe 0000:43:00.1: failed to load because an unsupported SFP+
or QSFP module type was detected.
[ 4817.900178] ixgbe 0000:43:00.1: Reload the driver after installing a
supported module.
[ 4817.940318] ixgbe 0000:44:00.0: failed to load because an unsupported SFP+
or QSFP module type was detected.
[ 4817.951357] ixgbe 0000:44:00.0: Reload the driver after installing a
supported module.
root@ubuntu1404:~# modinfo ixgbe
filename:
/lib/modules/3.13.0-91-generic/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version: 4.3.15
license: GPL
description: Intel(R) 10 Gigabit PCI Express Network Driver
author: Intel Corporation, <[email protected]>
srcversion: 7AED484083B2C5B86424A3A
alias: pci:v00008086d000015ADsv*sd*bc*sc*i*
alias: pci:v00008086d000015ACsv*sd*bc*sc*i*
alias: pci:v00008086d000015ABsv*sd*bc*sc*i*
alias: pci:v00008086d000015AAsv*sd*bc*sc*i*
alias: pci:v00008086d000015D1sv*sd*bc*sc*i*
alias: pci:v00008086d00001563sv*sd*bc*sc*i*
alias: pci:v00008086d00001560sv*sd*bc*sc*i*
alias: pci:v00008086d00001558sv*sd*bc*sc*i*
alias: pci:v00008086d0000154Asv*sd*bc*sc*i*
alias: pci:v00008086d00001557sv*sd*bc*sc*i*
alias: pci:v00008086d0000154Fsv*sd*bc*sc*i*
alias: pci:v00008086d0000154Dsv*sd*bc*sc*i*
alias: pci:v00008086d00001528sv*sd*bc*sc*i*
alias: pci:v00008086d000010F8sv*sd*bc*sc*i*
alias: pci:v00008086d0000151Csv*sd*bc*sc*i*
alias: pci:v00008086d00001529sv*sd*bc*sc*i*
alias: pci:v00008086d0000152Asv*sd*bc*sc*i*
alias: pci:v00008086d000010F9sv*sd*bc*sc*i*
alias: pci:v00008086d00001514sv*sd*bc*sc*i*
alias: pci:v00008086d00001507sv*sd*bc*sc*i*
alias: pci:v00008086d000010FBsv*sd*bc*sc*i*
alias: pci:v00008086d00001517sv*sd*bc*sc*i*
alias: pci:v00008086d000010FCsv*sd*bc*sc*i*
alias: pci:v00008086d000010F7sv*sd*bc*sc*i*
alias: pci:v00008086d00001508sv*sd*bc*sc*i*
alias: pci:v00008086d000010DBsv*sd*bc*sc*i*
alias: pci:v00008086d000010F4sv*sd*bc*sc*i*
alias: pci:v00008086d000010E1sv*sd*bc*sc*i*
alias: pci:v00008086d000010F1sv*sd*bc*sc*i*
alias: pci:v00008086d000010ECsv*sd*bc*sc*i*
alias: pci:v00008086d000010DDsv*sd*bc*sc*i*
alias: pci:v00008086d0000150Bsv*sd*bc*sc*i*
alias: pci:v00008086d000010C8sv*sd*bc*sc*i*
alias: pci:v00008086d000010C7sv*sd*bc*sc*i*
alias: pci:v00008086d000010C6sv*sd*bc*sc*i*
alias: pci:v00008086d000010B6sv*sd*bc*sc*i*
depends: ptp,dca,vxlan
vermagic: 3.13.0-91-generic SMP mod_unload modversions
parm: InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X),
default IntMode (deprecated) (array of int)
parm: IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X),
default 2 (array of int)
parm: MQ:Disable or enable Multiple Queues, default 1 (array of int)
parm: DCA:Disable or enable Direct Cache Access, 0=disabled,
1=descriptor only, 2=descriptor and data (array of int)
parm: RSS:Number of Receive-Side Scaling Descriptor Queues, default
0=number of cpus (array of int)
parm: VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable,
2-16 enable (default=8) (array of int)
parm: max_vfs:Number of Virtual Functions: 0 = disable (default),
1-63 = enable this many VFs (array of int)
parm: VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of
int)
parm: InterruptThrottleRate:Maximum interrupts per second, per
vector, (0,1,956-488281), default 1 (array of int)
parm: LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)
parm: LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of
int)
parm: LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of
int)
parm: LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of
int)
parm: LLIVLANP:Low Latency Interrupt on VLAN priority threshold
(array of int)
parm: FdirPballoc:Flow Director packet buffer allocation level:
1 = 8k hash filters or 2k perfect filters
2 = 16k hash filters or 4k perfect filters
3 = 32k hash filters or 8k perfect filters (array of
int)
parm: AtrSampleRate:Software ATR Tx packet sample rate (array of int)
parm: FCoE:Disable or enable FCoE Offload, default 1 (array of int)
parm: MDD:Malicious Driver Detection: (0,1), default 1 = on (array of
int)
parm: LRO:Large Receive Offload (0,1), default 0 = off (array of int)
parm: allow_unsupported_sfp:Allow unsupported and untested SFP+
modules on 82599 based adapters, default 0 = Disable (array of int)
parm: dmac_watchdog:DMA coalescing watchdog in microseconds
(0,41-10000), default 0 = off (array of int)
parm: vxlan_rx:VXLAN receive checksum offload (0,1), default 1 =
Enable (array of int)
root@ubuntu1404:~# modprobe vfio-pci
00.0/sriov_numvfs
sriov_numvfssys/devices/pci0000:00/0000:00:01.0/0000:01:00.1/sriov_numvfsroot@ubuntu1404:~#
echo "1" > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/
sriov_numvfs404:~# echo "1" >
/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.1/
root@ubuntu1404:~# resize
COLUMNS=211;
LINES=53;
export COLUMNS LINES;
root@ubuntu1404:~# ip link set dev eth1 vf 0 spoofchk off
et dev eth1 vf 0 mac de:ad:be:ef:00:00
ip link set dev eth9 vf 0 mac de:ad:be:ef:00:01
root@ubuntu1404:~# ip link set dev eth9 vf 0 spoofchk off
root@ubuntu1404:~# ip link set dev eth1 vf 0 mac de:ad:be:ef:00:00
root@ubuntu1404:~# ip link set dev eth9 vf 0 mac de:ad:be:ef:00:01
root@ubuntu1404:~#
root@ubuntu1404:~# rmmod ixgbevf
root@ubuntu1404:~# modprobe ixgbevf
root@ubuntu1404:~# ip link set eth1 up
root@ubuntu1404:~# ip link set eth9 up
root@ubuntu1404:~#
root@ubuntu1404:~# bridge link set dev eth1 hwmode veb
root@ubuntu1404:~# bridge link set dev eth9 hwmode veb
root@ubuntu1404:~#
root@ubuntu1404:~# echo "8086 10ed" > /sys/bus/pci/drivers/vfio-pci/new_id
root@ubuntu1404:~# echo "0000:01:10.0" >
/sys/bus/pci/devices/0000\:01\:10.0/driver/unbind
root@ubuntu1404:~# echo "0000:01:10.1" >
/sys/bus/pci/devices/0000\:01\:10.1/driver/unbind
root@ubuntu1404:~# echo "0000:01:10.0" > /sys/bus/pci/drivers/vfio-pci/bind
root@ubuntu1404:~# echo "0000:01:10.1" > /sys/bus/pci/drivers/vfio-pci/bind
root@ubuntu1404:~#
root@ubuntu1404:~#
root@ubuntu1404:~# ethtool -l eth1
Channel parameters for eth1:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 1
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
root@ubuntu1404:~# ethtool -l eth9
Channel parameters for eth9:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 1
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
root@ubuntu1404:~# tc qdisc add dev eth1 root mqprio num_tc 8
root@ubuntu1404:~# tc qdisc add dev eth9 root mqprio num_tc 8
root@ubuntu1404:~#
root@ubuntu1404:~# taskset -c 0,2,26,4,28,6,30,8,32,10,34,12,36,14,38
qemu-system-x86_64 -enable-kvm -cpu host -smp 15 \
> -serial telnet::4444,server,nowait \
> -monitor telnet::5555,server,nowait \
> -device virtio-net,mac=DE:AD:DE:01:02:03,netdev=user.0 \
> -netdev user,id=user.0,dnssearch=vm.6wind.com,hostfwd=tcp::2222-:22 \
> -hda centos-7-template.qcow2 -m 6G \
> -device vfio-pci,host=01:10.0 -device vfio-pci,host=01:10.1 --nographic &
[1] 2851
root@ubuntu1404:~# [ 4917.696698] kvm [2851]: vcpu0 unhandled rdmsr: 0x1c9
[ 4917.702297] kvm [2851]: vcpu0 unhandled rdmsr: 0x1a6
[ 4917.707877] kvm [2851]: vcpu0 unhandled rdmsr: 0x1a7
[ 4917.713456] kvm [2851]: vcpu0 unhandled rdmsr: 0x3f6
[ 4918.731234] kvm [2851]: vcpu2 unhandled rdmsr: 0x606
[ 4922.596487] kvm [2851]: vcpu12 unhandled rdmsr: 0xd90
On the VM side
---------------
CentOS Linux 7 (Core)
Kernel 3.10.0-327.22.2.el7.x86_64 on an x86_64
centos7 login: root
Password:
Last login: Tue Sep 6 17:33:59 on ttyS0
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
3: ens5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
4: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
qlen 1000
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0
valid_lft 86256sec preferred_lft 86256sec
inet6 fe80::dcad:deff:fe01:203/64 scope link
valid_lft forever preferred_lft forever
[root@centos7 ~]# ip li set ens4 up
[ 162.228284] ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps
[root@centos7 ~]# ip li set ens5 up
[ 166.828243] ixgbevf 0000:00:05.0: NIC Link is Up 10 Gbps
[ 166.830297] IPv6: ADDRCONF(NETDEV_UP): ens5: link is not ready
[ 166.831105] IPv6: ADDRCONF(NETDEV_CHANGE): ens5: link becomes ready
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:0/64 scope link
valid_lft forever preferred_lft forever
3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:1/64 scope link
valid_lft forever preferred_lft forever
4: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
qlen 1000
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0
valid_lft 86238sec preferred_lft 86238sec
inet6 fe80::dcad:deff:fe01:203/64 scope link
valid_lft forever preferred_lft forever
[root@centos7 ~]# ip link add link ens4 name vlan1 type vlan id 1
[ 198.884719] 8021q: 802.1Q VLAN Support v1.8
[ 198.885423] 8021q: adding VLAN 0 to HW filter on device ens4
[ 198.886916] 8021q: adding VLAN 0 to HW filter on device ens5
[ 198.891302] 8021q: adding VLAN 0 to HW filter on device mgmt0
RTNETLINK answers: Permission denied
[root@centos7 ~]# ip link add link ens5 name vlan2 type vlan id 2
RTNETLINK answers: Permission denied
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:0/64 scope link
valid_lft forever preferred_lft forever
3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:1/64 scope link
valid_lft forever preferred_lft forever
4: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
qlen 1000
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0
valid_lft 86188sec preferred_lft 86188sec
inet6 fe80::dcad:deff:fe01:203/64 scope link
valid_lft forever preferred_lft forever
[root@centos7 ~]# tail -10 /var/log/messages
Sep 6 18:56:13 centos7 systemd: Started Session 1 of user root.
Sep 6 18:56:13 centos7 systemd: Starting Session 1 of user root.
Sep 6 18:56:39 centos7 kernel: ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps
Sep 6 18:56:43 centos7 kernel: ixgbevf 0000:00:05.0: NIC Link is Up 10 Gbps
Sep 6 18:56:43 centos7 kernel: IPv6: ADDRCONF(NETDEV_UP): ens5: link is not
ready
Sep 6 18:56:43 centos7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ens5: link
becomes ready
Sep 6 18:57:15 centos7 kernel: 8021q: 802.1Q VLAN Support v1.8
Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device ens4
Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device ens5
Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device
mgmt0
[root@centos7 ~]#
Even with an ubuntu-14.04 VM, issue can be reproduced:
root@ubuntu1404:~# modinfo ixgbevf
filename:
/lib/modules/3.13.0-92-generic/kernel/drivers/net/ixgbevf/ixgbevf.ko
version: 2.16.1
license: GPL
description: Intel(R) 10 Gigabit Virtual Function Network Driver
author: Intel Corporation, <[email protected]>
srcversion: BD04DA0615170C30891C96E
alias: pci:v00008086d00001515sv*sd*bc*sc*i*
alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
depends:
vermagic: 3.13.0-92-generic SMP mod_unload modversions
parm: InterruptThrottleRate:Maximum interrupts per second, per
vector, (956-488281, 0=off, 1=dynamic), default 1 (array of int)
root@ubuntu1404:~# uname -a
Linux ubuntu1404 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu1404:~# lsb_release -da
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
root@ubuntu1404:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
group default qlen 1000
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global mgmt0
valid_lft forever preferred_lft forever
inet6 fe80::dcad:deff:fe01:203/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group
default qlen 1000
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:0/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group
default qlen 1000
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:1/64 scope link
valid_lft forever preferred_lft forever
root@ubuntu1404:~# ip link add link eth1 name vlan1 type vlan id 1
RTNETLINK answers: Permission denied
root@ubuntu1404:~# ip link add link eth2 name vlan2 type vlan id 2
RTNETLINK answers: Permission denied
root@ubuntu1404:~#
Thus, the VLAN creation is not supported with VF queueuing enabled (8 combined
queues which is the maximum).
Yet, with default queues (without typing tc command), VLANs were properly
created.
** Affects: dkms (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1621474
Title:
VLAN creation fails with VF multi queueuing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/dkms/+bug/1621474/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
