With additional tests basing on the last driver versions on the host and guest: * 4.4.6 ixgbe version on the host * 3.2.2 ixgbevf version on the guest,
and with following kernel version: root@ubuntu1604:~# uname -a Linux ubuntu1604 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux root@ubuntu1604:~# lsb_release -da No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.1 LTS Release: 16.04 Codename: xenial on both host and VM, the creation of VLAN was not supported: root@ubuntu1604:~# ip link add link ens5 name vlan2 type vlan id 2 [ 3275.847268] VF could not set VLAN 2 <========= Thus, even with recent versions, issue can be reproduced. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1621474 Title: VLAN creation fails with VF multi queueuing Status in linux package in Ubuntu: Incomplete Bug description: Description ------------ It can be read in SR-IOV companion guide that 82599 chipsets : * The 82599 has 128 Transmit and 128 Receive queues. They are generally referred to/thought of as queue pairs (1 Transmit and 1 Receive queue). This gives the 82599 128 queue pairs. * The 82599 has up to 64 pools, with each pool having two queue pairs in it. That is two Transmit and Receive queue assigned to each VF. * The 82599 allows for a variable configuration for the number of pools. In SR-IOV mode, there can be 16, 32 or 64 pools. The last line suggest using combined queues of 2, 4 and 8. The only way, for now we found, is using tc on the host especially as the ethtool command is not supported (ethtool -L eth10 combined 8 for example). And, with following command "tc qdisc add dev eth10 root mqprio num_tc 8", we will be able to use up to 8 combined queues on the host PF and guests which are using VFs. Yet, with 8 combined queues with tc commnad, VLAN creation was not permitted. Following driver versions have been used on host and guest to enable this multiqueue option: * 4.3.15 ixgbe in the host * 3.1.2 ixgbevf in the guest To reproduce ------------- On the host side ---------------- root@ubuntu1404:~# rmmod ixgbe root@ubuntu1404:~# modprobe ixgbe [ 4817.027637] ixgbe 0000:05:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected. [ 4817.038630] ixgbe 0000:05:00.0: Reload the driver after installing a supported module. [ 4817.418360] ixgbe 0000:41:00.1: failed to load because an unsupported SFP+ or QSFP module type was detected. [ 4817.429354] ixgbe 0000:41:00.1: Reload the driver after installing a supported module. [ 4817.838086] ixgbe 0000:43:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected. [ 4817.849091] ixgbe 0000:43:00.0: Reload the driver after installing a supported module. [ 4817.889189] ixgbe 0000:43:00.1: failed to load because an unsupported SFP+ or QSFP module type was detected. [ 4817.900178] ixgbe 0000:43:00.1: Reload the driver after installing a supported module. [ 4817.940318] ixgbe 0000:44:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected. [ 4817.951357] ixgbe 0000:44:00.0: Reload the driver after installing a supported module. root@ubuntu1404:~# modinfo ixgbe filename: /lib/modules/3.13.0-91-generic/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko version: 4.3.15 license: GPL description: Intel(R) 10 Gigabit PCI Express Network Driver author: Intel Corporation, <linux.n...@intel.com> srcversion: 7AED484083B2C5B86424A3A alias: pci:v00008086d000015ADsv*sd*bc*sc*i* alias: pci:v00008086d000015ACsv*sd*bc*sc*i* alias: pci:v00008086d000015ABsv*sd*bc*sc*i* alias: pci:v00008086d000015AAsv*sd*bc*sc*i* alias: pci:v00008086d000015D1sv*sd*bc*sc*i* alias: pci:v00008086d00001563sv*sd*bc*sc*i* alias: pci:v00008086d00001560sv*sd*bc*sc*i* alias: pci:v00008086d00001558sv*sd*bc*sc*i* alias: pci:v00008086d0000154Asv*sd*bc*sc*i* alias: pci:v00008086d00001557sv*sd*bc*sc*i* alias: pci:v00008086d0000154Fsv*sd*bc*sc*i* alias: pci:v00008086d0000154Dsv*sd*bc*sc*i* alias: pci:v00008086d00001528sv*sd*bc*sc*i* alias: pci:v00008086d000010F8sv*sd*bc*sc*i* alias: pci:v00008086d0000151Csv*sd*bc*sc*i* alias: pci:v00008086d00001529sv*sd*bc*sc*i* alias: pci:v00008086d0000152Asv*sd*bc*sc*i* alias: pci:v00008086d000010F9sv*sd*bc*sc*i* alias: pci:v00008086d00001514sv*sd*bc*sc*i* alias: pci:v00008086d00001507sv*sd*bc*sc*i* alias: pci:v00008086d000010FBsv*sd*bc*sc*i* alias: pci:v00008086d00001517sv*sd*bc*sc*i* alias: pci:v00008086d000010FCsv*sd*bc*sc*i* alias: pci:v00008086d000010F7sv*sd*bc*sc*i* alias: pci:v00008086d00001508sv*sd*bc*sc*i* alias: pci:v00008086d000010DBsv*sd*bc*sc*i* alias: pci:v00008086d000010F4sv*sd*bc*sc*i* alias: pci:v00008086d000010E1sv*sd*bc*sc*i* alias: pci:v00008086d000010F1sv*sd*bc*sc*i* alias: pci:v00008086d000010ECsv*sd*bc*sc*i* alias: pci:v00008086d000010DDsv*sd*bc*sc*i* alias: pci:v00008086d0000150Bsv*sd*bc*sc*i* alias: pci:v00008086d000010C8sv*sd*bc*sc*i* alias: pci:v00008086d000010C7sv*sd*bc*sc*i* alias: pci:v00008086d000010C6sv*sd*bc*sc*i* alias: pci:v00008086d000010B6sv*sd*bc*sc*i* depends: ptp,dca,vxlan vermagic: 3.13.0-91-generic SMP mod_unload modversions parm: InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int) parm: IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int) parm: MQ:Disable or enable Multiple Queues, default 1 (array of int) parm: DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int) parm: RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int) parm: VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default=8) (array of int) parm: max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int) parm: VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of int) parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int) parm: LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int) parm: LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int) parm: LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int) parm: LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int) parm: LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int) parm: FdirPballoc:Flow Director packet buffer allocation level: 1 = 8k hash filters or 2k perfect filters 2 = 16k hash filters or 4k perfect filters 3 = 32k hash filters or 8k perfect filters (array of int) parm: AtrSampleRate:Software ATR Tx packet sample rate (array of int) parm: FCoE:Disable or enable FCoE Offload, default 1 (array of int) parm: MDD:Malicious Driver Detection: (0,1), default 1 = on (array of int) parm: LRO:Large Receive Offload (0,1), default 0 = off (array of int) parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int) parm: dmac_watchdog:DMA coalescing watchdog in microseconds (0,41-10000), default 0 = off (array of int) parm: vxlan_rx:VXLAN receive checksum offload (0,1), default 1 = Enable (array of int) root@ubuntu1404:~# modprobe vfio-pci 00.0/sriov_numvfs sriov_numvfssys/devices/pci0000:00/0000:00:01.0/0000:01:00.1/sriov_numvfsroot@ubuntu1404:~# echo "1" > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/ sriov_numvfs404:~# echo "1" > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.1/ root@ubuntu1404:~# resize COLUMNS=211; LINES=53; export COLUMNS LINES; root@ubuntu1404:~# ip link set dev eth1 vf 0 spoofchk off et dev eth1 vf 0 mac de:ad:be:ef:00:00 ip link set dev eth9 vf 0 mac de:ad:be:ef:00:01 root@ubuntu1404:~# ip link set dev eth9 vf 0 spoofchk off root@ubuntu1404:~# ip link set dev eth1 vf 0 mac de:ad:be:ef:00:00 root@ubuntu1404:~# ip link set dev eth9 vf 0 mac de:ad:be:ef:00:01 root@ubuntu1404:~# root@ubuntu1404:~# rmmod ixgbevf root@ubuntu1404:~# modprobe ixgbevf root@ubuntu1404:~# ip link set eth1 up root@ubuntu1404:~# ip link set eth9 up root@ubuntu1404:~# root@ubuntu1404:~# bridge link set dev eth1 hwmode veb root@ubuntu1404:~# bridge link set dev eth9 hwmode veb root@ubuntu1404:~# root@ubuntu1404:~# echo "8086 10ed" > /sys/bus/pci/drivers/vfio-pci/new_id root@ubuntu1404:~# echo "0000:01:10.0" > /sys/bus/pci/devices/0000\:01\:10.0/driver/unbind root@ubuntu1404:~# echo "0000:01:10.1" > /sys/bus/pci/devices/0000\:01\:10.1/driver/unbind root@ubuntu1404:~# echo "0000:01:10.0" > /sys/bus/pci/drivers/vfio-pci/bind root@ubuntu1404:~# echo "0000:01:10.1" > /sys/bus/pci/drivers/vfio-pci/bind root@ubuntu1404:~# root@ubuntu1404:~# root@ubuntu1404:~# ethtool -l eth1 Channel parameters for eth1: Pre-set maximums: RX: 0 TX: 0 Other: 1 Combined: 1 Current hardware settings: RX: 0 TX: 0 Other: 1 Combined: 4 root@ubuntu1404:~# ethtool -l eth9 Channel parameters for eth9: Pre-set maximums: RX: 0 TX: 0 Other: 1 Combined: 1 Current hardware settings: RX: 0 TX: 0 Other: 1 Combined: 4 root@ubuntu1404:~# tc qdisc add dev eth1 root mqprio num_tc 8 root@ubuntu1404:~# tc qdisc add dev eth9 root mqprio num_tc 8 root@ubuntu1404:~# root@ubuntu1404:~# taskset -c 0,2,26,4,28,6,30,8,32,10,34,12,36,14,38 qemu-system-x86_64 -enable-kvm -cpu host -smp 15 \ > -serial telnet::4444,server,nowait \ > -monitor telnet::5555,server,nowait \ > -device virtio-net,mac=DE:AD:DE:01:02:03,netdev=user.0 \ > -netdev user,id=user.0,dnssearch=vm.6wind.com,hostfwd=tcp::2222-:22 \ > -hda centos-7-template.qcow2 -m 6G \ > -device vfio-pci,host=01:10.0 -device vfio-pci,host=01:10.1 --nographic & [1] 2851 root@ubuntu1404:~# [ 4917.696698] kvm [2851]: vcpu0 unhandled rdmsr: 0x1c9 [ 4917.702297] kvm [2851]: vcpu0 unhandled rdmsr: 0x1a6 [ 4917.707877] kvm [2851]: vcpu0 unhandled rdmsr: 0x1a7 [ 4917.713456] kvm [2851]: vcpu0 unhandled rdmsr: 0x3f6 [ 4918.731234] kvm [2851]: vcpu2 unhandled rdmsr: 0x606 [ 4922.596487] kvm [2851]: vcpu12 unhandled rdmsr: 0xd90 On the VM side --------------- CentOS Linux 7 (Core) Kernel 3.10.0-327.22.2.el7.x86_64 on an x86_64 centos7 login: root Password: Last login: Tue Sep 6 17:33:59 on ttyS0 [root@centos7 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff 3: ens5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff 4: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0 valid_lft 86256sec preferred_lft 86256sec inet6 fe80::dcad:deff:fe01:203/64 scope link valid_lft forever preferred_lft forever [root@centos7 ~]# ip li set ens4 up [ 162.228284] ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps [root@centos7 ~]# ip li set ens5 up [ 166.828243] ixgbevf 0000:00:05.0: NIC Link is Up 10 Gbps [ 166.830297] IPv6: ADDRCONF(NETDEV_UP): ens5: link is not ready [ 166.831105] IPv6: ADDRCONF(NETDEV_CHANGE): ens5: link becomes ready [root@centos7 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff inet6 fe80::dcad:beff:feef:0/64 scope link valid_lft forever preferred_lft forever 3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff inet6 fe80::dcad:beff:feef:1/64 scope link valid_lft forever preferred_lft forever 4: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0 valid_lft 86238sec preferred_lft 86238sec inet6 fe80::dcad:deff:fe01:203/64 scope link valid_lft forever preferred_lft forever [root@centos7 ~]# ip link add link ens4 name vlan1 type vlan id 1 [ 198.884719] 8021q: 802.1Q VLAN Support v1.8 [ 198.885423] 8021q: adding VLAN 0 to HW filter on device ens4 [ 198.886916] 8021q: adding VLAN 0 to HW filter on device ens5 [ 198.891302] 8021q: adding VLAN 0 to HW filter on device mgmt0 RTNETLINK answers: Permission denied [root@centos7 ~]# ip link add link ens5 name vlan2 type vlan id 2 RTNETLINK answers: Permission denied [root@centos7 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff inet6 fe80::dcad:beff:feef:0/64 scope link valid_lft forever preferred_lft forever 3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff inet6 fe80::dcad:beff:feef:1/64 scope link valid_lft forever preferred_lft forever 4: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0 valid_lft 86188sec preferred_lft 86188sec inet6 fe80::dcad:deff:fe01:203/64 scope link valid_lft forever preferred_lft forever [root@centos7 ~]# tail -10 /var/log/messages Sep 6 18:56:13 centos7 systemd: Started Session 1 of user root. Sep 6 18:56:13 centos7 systemd: Starting Session 1 of user root. Sep 6 18:56:39 centos7 kernel: ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps Sep 6 18:56:43 centos7 kernel: ixgbevf 0000:00:05.0: NIC Link is Up 10 Gbps Sep 6 18:56:43 centos7 kernel: IPv6: ADDRCONF(NETDEV_UP): ens5: link is not ready Sep 6 18:56:43 centos7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ens5: link becomes ready Sep 6 18:57:15 centos7 kernel: 8021q: 802.1Q VLAN Support v1.8 Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device ens4 Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device ens5 Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device mgmt0 [root@centos7 ~]# Even with an ubuntu-14.04 VM, issue can be reproduced: root@ubuntu1404:~# modinfo ixgbevf filename: /lib/modules/3.13.0-92-generic/kernel/drivers/net/ixgbevf/ixgbevf.ko version: 2.16.1 license: GPL description: Intel(R) 10 Gigabit Virtual Function Network Driver author: Intel Corporation, <linux.n...@intel.com> srcversion: BD04DA0615170C30891C96E alias: pci:v00008086d00001515sv*sd*bc*sc*i* alias: pci:v00008086d000010EDsv*sd*bc*sc*i* depends: vermagic: 3.13.0-92-generic SMP mod_unload modversions parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (956-488281, 0=off, 1=dynamic), default 1 (array of int) root@ubuntu1404:~# uname -a Linux ubuntu1404 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux root@ubuntu1404:~# lsb_release -da No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.4 LTS Release: 14.04 Codename: trusty root@ubuntu1404:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global mgmt0 valid_lft forever preferred_lft forever inet6 fe80::dcad:deff:fe01:203/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff inet6 fe80::dcad:beff:feef:0/64 scope link valid_lft forever preferred_lft forever 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff inet6 fe80::dcad:beff:feef:1/64 scope link valid_lft forever preferred_lft forever root@ubuntu1404:~# ip link add link eth1 name vlan1 type vlan id 1 RTNETLINK answers: Permission denied root@ubuntu1404:~# ip link add link eth2 name vlan2 type vlan id 2 RTNETLINK answers: Permission denied root@ubuntu1404:~# Thus, the VLAN creation is not supported with VF queueuing enabled (8 combined queues which is the maximum). Yet, with default queues (without typing tc command), VLANs were properly created. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1621474/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
