[Kernel-packages] [Bug 2044810] Re: VF cannot creation with large CPU core systems when RDMA enabled with intel ice driver
Someone can take a look for this issue, thanks :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-hwe-5.15 in Ubuntu. https://bugs.launchpad.net/bugs/2044810 Title: VF cannot creation with large CPU core systems when RDMA enabled with intel ice driver Status in linux-hwe-5.15 package in Ubuntu: New Bug description: Issue Environment: == root@npx:~# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; UBUNTU_CODENAME=jammy root@npx:~# uname -r 5.15.0-88-generic root@npx:~# lscpu | head -n 5 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 256 root@npx:~# ethtool -i ens2f0 driver: ice version: 5.15.0-88-generic firmware-version: 4.40 0x8001c7d5 1.3534.0 expansion-rom-version: bus-info: :16:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes root@npx:~# lspci -s 16:00.0 -vvv 16:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02) Subsystem: Intel Corporation Ethernet Network Adapter E810-XXV-4 Physical Slot: 2 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [210 v1] Physical Layer 16.0 GT/s Capabilities: [250 v1] Lane Margining at the Receiver Kernel driver in use: ice Kernel modules: ice Issue Description: == # echo 1 > /sys/class/net/ens2f0/device/sriov_numvfs [ 5734.469217] ice :16:00.0: Enabling 1 VFs [ 5734.574945] pci :16:01.0: [8086:1889] type 00 class 0x02 [ 5734.574970] pci :16:01.0: enabling Extended Tags [ 5734.575471] pci :16:01.0: Adding to iommu group 443 [ 5734.575718] ice :16:00.0: Only 0 MSI-X interrupts available for SR-IOV. Not enough to support minimum of 2 MSI-X interrupts per VF for 1 VFs [ 5734.575815] ice :16:00.0: Not enough resources for 1 VFs, try with fewer number of VFs [ 5734.576861] pci :16:01.0: Removing from iommu group 443 [ 5734.623292] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver [ 5734.623297] Copyright (c) 2013 - 2018 Intel Corporation. [ 5735.598871] ice :16:00.0: Failed to enable SR-IOV: -28 Issue Found: 1> After disable RDMA, the VF creation works fine; from kernel code, the MSIx are preserved by LAN and RDMA based on CPU cores, this will exhauste all available MSIx for larger core systems (some PF port will only have 512 MSIx in total), this doesn't make sense as the default value (at least make sure a few number VFs can be created successfully if NIC support it) 2> When do the MSIx resource reallocation manually, still raise below error, this is some what a strange behavior, it's better to allow such actions by default from kernel: root@npx:~# devlink resource show pci/:16:00.0 kernel answers: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.15/+bug/2044810/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2044810] Re: VF cannot creation with large CPU core systems when RDMA enabled with intel ice driver
** Package changed: ubuntu => linux-hwe-5.15 (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-hwe-5.15 in Ubuntu. https://bugs.launchpad.net/bugs/2044810 Title: VF cannot creation with large CPU core systems when RDMA enabled with intel ice driver Status in linux-hwe-5.15 package in Ubuntu: New Bug description: Issue Environment: == root@npx:~# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; UBUNTU_CODENAME=jammy root@npx:~# uname -r 5.15.0-88-generic root@npx:~# lscpu | head -n 5 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 256 root@npx:~# ethtool -i ens2f0 driver: ice version: 5.15.0-88-generic firmware-version: 4.40 0x8001c7d5 1.3534.0 expansion-rom-version: bus-info: :16:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes root@npx:~# lspci -s 16:00.0 -vvv 16:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02) Subsystem: Intel Corporation Ethernet Network Adapter E810-XXV-4 Physical Slot: 2 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [210 v1] Physical Layer 16.0 GT/s Capabilities: [250 v1] Lane Margining at the Receiver Kernel driver in use: ice Kernel modules: ice Issue Description: == # echo 1 > /sys/class/net/ens2f0/device/sriov_numvfs [ 5734.469217] ice :16:00.0: Enabling 1 VFs [ 5734.574945] pci :16:01.0: [8086:1889] type 00 class 0x02 [ 5734.574970] pci :16:01.0: enabling Extended Tags [ 5734.575471] pci :16:01.0: Adding to iommu group 443 [ 5734.575718] ice :16:00.0: Only 0 MSI-X interrupts available for SR-IOV. Not enough to support minimum of 2 MSI-X interrupts per VF for 1 VFs [ 5734.575815] ice :16:00.0: Not enough resources for 1 VFs, try with fewer number of VFs [ 5734.576861] pci :16:01.0: Removing from iommu group 443 [ 5734.623292] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver [ 5734.623297] Copyright (c) 2013 - 2018 Intel Corporation. [ 5735.598871] ice :16:00.0: Failed to enable SR-IOV: -28 Issue Found: 1> After disable RDMA, the VF creation works fine; from kernel code, the MSIx are preserved by LAN and RDMA based on CPU cores, this will exhauste all available MSIx for larger core systems (some PF port will only have 512 MSIx in total), this doesn't make sense as the default value (at least make sure a few number VFs can be created successfully if NIC support it) 2> When do the MSIx resource reallocation manually, still raise below error, this is some what a strange behavior, it's better to allow such actions by default from kernel: root@npx:~# devlink resource show pci/:16:00.0 kernel answers: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.15/+bug/2044810/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
