[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
** Description changed: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. The following conditions are required to trigger the bug: - A port VLAN must be assigned by the host - The MTU must be set >1500 by the guest It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. [ Impact ] Intel E810 (host: ice) SR-IOV VFs (guest: iavf) fail when a port VLAN is - assigned by the host and the guest changes the MTU >1500. The guest iavf driver - logs the following error and traffic flow stops: + assigned by the host and the guest changes the MTU >1500. The guest iavf + driver logs the following error and traffic flow stops: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 - There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be used - without this patch. - + There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be + used without this patch. [ Fix ] During queue configuration, VF's max packet size was set to IAVF_MAX_RXBUFFER but on ice max frame size was smaller by VLAN_HLEN due to making some space for port VLAN as VF is not aware whether it's in a port VLAN. This mismatch in sizes caused ice to reject queue configuration with ERR_PARAM error. Proper max_mtu is sent from ice PF to VF with GET_VF_RESOURCES msg but VF does not look at this. In iavf change max_frame from IAVF_MAX_RXBUFFER to max_mtu received from pf with GET_VF_RESOURCES msg to make vf's max_frame_size dependent from pf. Add check if received max_mtu is not in eligible range then set it to IAVF_MAX_RXBUFFER. - [ Test Plan ] 1. Create a focal VM and assign an Intel E810 (ice) SR-IOV VF with a port vlan Openstack works, as does creating a VM directly with uvtool/libvirt. The syntax inside for the port VLAN is 2. Set an IP on the interface, set it’s link up and verify operation sudo ip a a 192.168.1.7/24 dev enp7s0 sudo ip link set up dev enp7s0 ping 192.168.1.1 3. Set the MTU to 9000 sudo ip link set mtu 9000 dev enp7s0 4. Test if the interface still works and whether the error was observed: iavf :07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 - [ Where problems could occur ] The iavf driver is unified for multiple host drivers including ice and i40e. - A companion patch for i40e was also submitted as it did not set the max_mtu - although the ice driver already did. We should test the fixed iavf driver - against both i40e (Intel E7xx) and ice (Intel E8xx) cards with and without that - patch. + A companion patch for i40e was also submitted as it did not set the + max_mtu although the ice driver already did. We should test the fixed iavf + driver against both i40e (Intel E7xx) and ice (Intel E8xx) cards with and + without that patch on the host kernel. https://lore.kernel.org/netdev/20220919223428.572091-4-anthony.l.ngu...@intel.com/ - [ Other Info ] Patch was developed by intel, has been accepted into v6.0-rc7 and already released into upstream stable v5.4.215, v5.15.71 and v5.19.12. - Early SRU is requested as the functionality is required now and there is no - workaround. + Early SRU is requested as the functionality is required now and there is + no workaround. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed ** Changed in: linux (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Confirmed Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
** Description changed: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. + There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d + claiming to resolve this error in some cases. It is already included in + the test kernel v5.15.0-41 and did not resolve the issue. - There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. + The following conditions are required to trigger the bug: + - A port VLAN must be assigned by the host + - The MTU must be set >1500 by the guest - These Virtual Machines do work with the Mainline v5.19 build on the host - and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related - commits that are not currently backported to v5.15 or any upstream - stable kernel: + It appears that this ice/iavf driver is quite immature as many + significant SR-IOV related fixes have landed in each of the recent + kernel releases and we may need to consider pro-actively backporting + more fixes. - 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] - be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] + [ Impact ] - Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: - efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] + Intel E810 (host: ice) SR-IOV VFs (guest: iavf) fail when a port VLAN is + assigned by the host and the guest changes the MTU >1500. The guest iavf driver + logs the following error and traffic flow stops: + iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 + + There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be used + without this patch. - It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. + [ Fix ] + + During queue configuration, VF's max packet size was set to + IAVF_MAX_RXBUFFER but on ice max frame size was smaller by VLAN_HLEN + due to making some space for port VLAN as VF is not aware whether it's + in a port VLAN. This mismatch in sizes caused ice to reject queue + configuration with ERR_PARAM error. Proper max_mtu is sent from ice PF + to VF with GET_VF_RESOURCES msg but VF does not look at this. + + In iavf change max_frame from IAVF_MAX_RXBUFFER to max_mtu + received from pf with GET_VF_RESOURCES msg to make vf's + max_frame_size dependent from pf. Add check if received max_mtu is + not in eligible range then set it to IAVF_MAX_RXBUFFER. + + + [ Test Plan ] + + 1. Create a focal VM and assign an Intel E810 (ice) SR-IOV VF with a port + vlan + + Openstack works, as does creating a VM directly with uvtool/libvirt. + + The syntax inside for the port VLAN is + + 2. Set an IP on the interface, set it’s link up and verify operation + + sudo ip a a 192.168.1.7/24 dev enp7s0 + sudo ip link set up dev enp7s0 + ping 192.168.1.1 + + 3. Set the MTU to 9000 + + sudo ip link set mtu 9000 dev enp7s0 + + 4. Test if the interface still works and whether the error was observed: + iavf :07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 + + + [ Where problems could occur ] + + The iavf driver is unified for multiple host drivers including ice and + i40e. + + A companion patch for i40e was also submitted as it did not set the max_mtu + although the ice driver already did. We should test the fixed iavf driver + against both i40e (Intel E7xx) and ice (Intel E8xx) cards with and without that + patch. + + https://lore.kernel.org/netdev/20220919223428.572091-4-anthony.l.ngu...@intel.com/ + + + [ Other Info ] + + Patch was developed by intel, has been accepted into v6.0-rc7 and already + released into upstream stable v5.4.215, v5.15.71 and v5.19.12. + + Early SRU is requested as the functionality is required now and there is no + workaround.
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
For users of i40e, seems this depends on the following host patch where the max_mtu from the host PF that is now used was not set on i40e (but was for ice). https://github.com/torvalds/linux/commit/372539def2824c43b6afe2403045b140f65c5acc It's also queued for stable. We may want to pull this at the same time. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
The patch has now made it upstream into the v6.0-rc7 release: https://github.com/torvalds/linux/commit/399c98c4dc50b7eb7e9f24da7ffdda6f025676ef It has also been pushed to the stable-queue for the upstream 5.4 and 5.15 kernels: https://www.spinics.net/lists/stable-commits/msg266383.html https://www.spinics.net/lists/stable-commits/msg266285.html A test kernel for Focal 5.4 and 5.15 was also built and tested. It does resolve the issue as expected. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
Been working with Trent and Intel, and below [1] is the patch for the MTU fix My test kernels are available in PPA [2] for both 5.4 and 5.15 [1] https://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue.git/commit/drivers/net/ethernet/intel/iavf?h=dev-queue=399c98c4dc50b7eb7e9f24da7ffdda6f025676ef [2] https://launchpad.net/~arif-ali/+archive/ubuntu/sf00343742 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
I have created a test kernel with the patches listed in comment #4 and comment #6. It does not include the mtu fix from comment #5 https://people.canonical.com/~mreed/misc/lp_1983656_e810_sriov/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
I may have missed this one... I meant to add it if it's not already picked up in a different commit: ice: Fix not stopping Tx queues for VFs https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/drivers/net/ethernet/intel/ice?id=b385cca47363316c6d9a74ae9db407bbc281f815 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
I was unable to locate the patch from comment #5 in linux-next or linus's tree as of yet. This patch may require a separate PR. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
The following patches are already applied to the jammy master-next kernel (Ubuntu-5.15.0-48.54-2-gd250073d5172) 5951a2b9812d v5.16-rc6 iavf: Fix VLAN feature flags after VFR e6ba5273d4ed v5.16-rc6 ice: Fix race conditions between virtchnl handling and VF ndo ops b385cca47363 v5.16-rc6 ice: Fix not stopping Tx queues for VFs 1a8c7778bcde v5.16-rc6 ice: Fix VF true promiscuous mode I was able to apply (0299faeaf8eb v5.16-rc6 ice: Remove toggling of antispoof for VF trusted promiscuous model) after applying the following patch: 29e71f41e7d2f7069c12c686ca4d222e8be2a2ee ice: Remove boolean vlan_promisc flag from function Are the three patches in the description still necessary? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
additional patch to pick up that is supposed to help address the issue with MTUs ice: Fix interface being down after reset with link-down-on-close flag on https://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue.git/commit/drivers/net/ethernet/intel?h=dev-queue=a2aa0cec547579cc137970c772ea82cbf5ee2608 So lets see if we can pull this in too, either with these patches or in a separate PR -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
Update from upstream: Please check out the other patches below that were sent together with the one patch I listed in my previous reply. Cherry-picking patches can be challenging with ice driver from later kernels as new features and bug fixes are sent continuously in each kernel and the dependency can be tricky. 5951a2b9812dv5.16-rc6 iavf: Fix VLAN feature flags after VFR e6ba5273d4edv5.16-rc6 ice: Fix race conditions between virtchnl handling and VF ndo ops b385cca47363v5.16-rc6 ice: Fix not stopping Tx queues for VFs 0299faeaf8ebv5.16-rc6 ice: Remove toggling of antispoof for VF trusted promiscuous mode 1a8c7778bcdev5.16-rc6 ice: Fix VF true promiscuous mode Regarding the issue of changing MTU, it is a known one in ice driver and the team is actively working on a fix. I will share once it is ready to be sent Linux upstream. Another note, for E810 NICs, please recommend customers to update the NVM image on the NIC to the latest one if possible. Here is the latest one: https://www.intel.com/content/www/us/en/download/19626/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapters-e810-series-linux.html -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
Updaet: We found that this bug was only triggering when you try to change the MTU to anything other than 1500. With MTU=1500 the VMs work fine even with the current 5.15 kernel. With MTU=1550 or MTU=9000 it fails even with the upstream 5.19 kernel and these patches. So seems we need to report and work with Intel on this to develop a new patch. Though probably we still need to backport these and maybe other ice patches to 5.15 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
Also, this will be resolved in Kinetic via 5.19 and Jammy 22.04.2 via HWE. ** Tags added: servcert-472 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1983656] Re: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf)
Hi Trent, What is the expectation here? To update 5.15? Or also 5.4? ** Changed in: linux (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice :98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf :00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel: 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
