[Kernel-packages] [Bug 2044810] Re: VF cannot creation with large CPU core systems when RDMA enabled with intel ice driver
Someone can take a look for this issue, thanks :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-hwe-5.15 in Ubuntu. https://bugs.launchpad.net/bugs/2044810 Title: VF cannot creation with large CPU core systems when RDMA enabled with intel ice driver Status in linux-hwe-5.15 package in Ubuntu: New Bug description: Issue Environment: == root@npx:~# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; UBUNTU_CODENAME=jammy root@npx:~# uname -r 5.15.0-88-generic root@npx:~# lscpu | head -n 5 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 256 root@npx:~# ethtool -i ens2f0 driver: ice version: 5.15.0-88-generic firmware-version: 4.40 0x8001c7d5 1.3534.0 expansion-rom-version: bus-info: :16:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes root@npx:~# lspci -s 16:00.0 -vvv 16:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02) Subsystem: Intel Corporation Ethernet Network Adapter E810-XXV-4 Physical Slot: 2 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [210 v1] Physical Layer 16.0 GT/s Capabilities: [250 v1] Lane Margining at the Receiver Kernel driver in use: ice Kernel modules: ice Issue Description: == # echo 1 > /sys/class/net/ens2f0/device/sriov_numvfs [ 5734.469217] ice :16:00.0: Enabling 1 VFs [ 5734.574945] pci :16:01.0: [8086:1889] type 00 class 0x02 [ 5734.574970] pci :16:01.0: enabling Extended Tags [ 5734.575471] pci :16:01.0: Adding to iommu group 443 [ 5734.575718] ice :16:00.0: Only 0 MSI-X interrupts available for SR-IOV. Not enough to support minimum of 2 MSI-X interrupts per VF for 1 VFs [ 5734.575815] ice :16:00.0: Not enough resources for 1 VFs, try with fewer number of VFs [ 5734.576861] pci :16:01.0: Removing from iommu group 443 [ 5734.623292] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver [ 5734.623297] Copyright (c) 2013 - 2018 Intel Corporation. [ 5735.598871] ice :16:00.0: Failed to enable SR-IOV: -28 Issue Found: 1> After disable RDMA, the VF creation works fine; from kernel code, the MSIx are preserved by LAN and RDMA based on CPU cores, this will exhauste all available MSIx for larger core systems (some PF port will only have 512 MSIx in total), this doesn't make sense as the default value (at least make sure a few number VFs can be created successfully if NIC support it) 2> When do the MSIx resource reallocation manually, still raise below error, this is some what a strange behavior, it's better to allow such actions by default from kernel: root@npx:~# devlink resource show pci/:16:00.0 kernel answers: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.15/+bug/2044810/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2044810] Re: VF cannot creation with large CPU core systems when RDMA enabled with intel ice driver
** Package changed: ubuntu => linux-hwe-5.15 (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-hwe-5.15 in Ubuntu. https://bugs.launchpad.net/bugs/2044810 Title: VF cannot creation with large CPU core systems when RDMA enabled with intel ice driver Status in linux-hwe-5.15 package in Ubuntu: New Bug description: Issue Environment: == root@npx:~# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; UBUNTU_CODENAME=jammy root@npx:~# uname -r 5.15.0-88-generic root@npx:~# lscpu | head -n 5 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 256 root@npx:~# ethtool -i ens2f0 driver: ice version: 5.15.0-88-generic firmware-version: 4.40 0x8001c7d5 1.3534.0 expansion-rom-version: bus-info: :16:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes root@npx:~# lspci -s 16:00.0 -vvv 16:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02) Subsystem: Intel Corporation Ethernet Network Adapter E810-XXV-4 Physical Slot: 2 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [210 v1] Physical Layer 16.0 GT/s Capabilities: [250 v1] Lane Margining at the Receiver Kernel driver in use: ice Kernel modules: ice Issue Description: == # echo 1 > /sys/class/net/ens2f0/device/sriov_numvfs [ 5734.469217] ice :16:00.0: Enabling 1 VFs [ 5734.574945] pci :16:01.0: [8086:1889] type 00 class 0x02 [ 5734.574970] pci :16:01.0: enabling Extended Tags [ 5734.575471] pci :16:01.0: Adding to iommu group 443 [ 5734.575718] ice :16:00.0: Only 0 MSI-X interrupts available for SR-IOV. Not enough to support minimum of 2 MSI-X interrupts per VF for 1 VFs [ 5734.575815] ice :16:00.0: Not enough resources for 1 VFs, try with fewer number of VFs [ 5734.576861] pci :16:01.0: Removing from iommu group 443 [ 5734.623292] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver [ 5734.623297] Copyright (c) 2013 - 2018 Intel Corporation. [ 5735.598871] ice :16:00.0: Failed to enable SR-IOV: -28 Issue Found: 1> After disable RDMA, the VF creation works fine; from kernel code, the MSIx are preserved by LAN and RDMA based on CPU cores, this will exhauste all available MSIx for larger core systems (some PF port will only have 512 MSIx in total), this doesn't make sense as the default value (at least make sure a few number VFs can be created successfully if NIC support it) 2> When do the MSIx resource reallocation manually, still raise below error, this is some what a strange behavior, it's better to allow such actions by default from kernel: root@npx:~# devlink resource show pci/:16:00.0 kernel answers: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.15/+bug/2044810/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
** Tags removed: verification-needed-focal-linux-aws-5.15 ** Tags added: verification-done-focal-linux-aws-5.15 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: Fix Released Status in linux source package in Kinetic: Fix Released Bug description: SRU Justification: [Impact] There is a user reporting errors in setup with their Intel E810 NIC with error messages saying that the driver cannot allocate enough MSI-X vectors on their 256 cpu-count system. It seems the ICE ethernet driver has an all or nothing approach to allocating MSI-X vectors and could request more MSI-X vectors than it finds available, which could lead to the driver failing to initialize and start. [Fix] The patch that fixes this allocates as many MSI-X vectors as it can to continue functionality by reducing the number of requested MSI-X vectors if it does not have enough to do full allocation. [Backport] In Jammy we do not carry patches for switchdev support in the driver so do not allocate the switchdev MSI-X vector for it. Also in Jammy use the older way of checking RDMA support by testing the RDMA bit is set as opposed to the newer ice_is_rdma_ena that the patch uses. [Test Plan] Install and startup Ice driver with an Intel 800 series NIC and check that we do not have the failure: Not enough device MSI-X vectors, requested = 260, available = 253 and check that everything works as expected. The backported patch for Jammy has been tested by the original user who submited the bug report with their high cpu count system and confirmed no errors. [Where problems could occur] There could be problems with the logic of reducing the MSI-X vector usage leading to more errors in the driver, but otherwise minimal regression potential as the code is mostly refactoring initial MSI-X setup. -- System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
** Tags removed: verification-needed-jammy ** Tags added: verification-done-jammy -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: Fix Released Status in linux source package in Kinetic: Fix Released Bug description: SRU Justification: [Impact] There is a user reporting errors in setup with their Intel E810 NIC with error messages saying that the driver cannot allocate enough MSI-X vectors on their 256 cpu-count system. It seems the ICE ethernet driver has an all or nothing approach to allocating MSI-X vectors and could request more MSI-X vectors than it finds available, which could lead to the driver failing to initialize and start. [Fix] The patch that fixes this allocates as many MSI-X vectors as it can to continue functionality by reducing the number of requested MSI-X vectors if it does not have enough to do full allocation. [Backport] In Jammy we do not carry patches for switchdev support in the driver so do not allocate the switchdev MSI-X vector for it. Also in Jammy use the older way of checking RDMA support by testing the RDMA bit is set as opposed to the newer ice_is_rdma_ena that the patch uses. [Test Plan] Install and startup Ice driver with an Intel 800 series NIC and check that we do not have the failure: Not enough device MSI-X vectors, requested = 260, available = 253 and check that everything works as expected. The backported patch for Jammy has been tested by the original user who submited the bug report with their high cpu count system and confirmed no errors. [Where problems could occur] There could be problems with the logic of reducing the MSI-X vector usage leading to more errors in the driver, but otherwise minimal regression potential as the code is mostly refactoring initial MSI-X setup. -- System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
** Tags removed: verification-needed-jammy ** Tags added: verification-done-jammy -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: Fix Released Status in linux source package in Kinetic: Fix Released Bug description: SRU Justification: [Impact] There is a user reporting errors in setup with their Intel E810 NIC with error messages saying that the driver cannot allocate enough MSI-X vectors on their 256 cpu-count system. It seems the ICE ethernet driver has an all or nothing approach to allocating MSI-X vectors and could request more MSI-X vectors than it finds available, which could lead to the driver failing to initialize and start. [Fix] The patch that fixes this allocates as many MSI-X vectors as it can to continue functionality by reducing the number of requested MSI-X vectors if it does not have enough to do full allocation. [Backport] In Jammy we do not carry patches for switchdev support in the driver so do not allocate the switchdev MSI-X vector for it. Also in Jammy use the older way of checking RDMA support by testing the RDMA bit is set as opposed to the newer ice_is_rdma_ena that the patch uses. [Test Plan] Install and startup Ice driver with an Intel 800 series NIC and check that we do not have the failure: Not enough device MSI-X vectors, requested = 260, available = 253 and check that everything works as expected. The backported patch for Jammy has been tested by the original user who submited the bug report with their high cpu count system and confirmed no errors. [Where problems could occur] There could be problems with the logic of reducing the MSI-X vector usage leading to more errors in the driver, but otherwise minimal regression potential as the code is mostly refactoring initial MSI-X setup. -- System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: Fix Committed Status in linux source package in Kinetic: Fix Committed Bug description: SRU Justification: [Impact] There is a user reporting errors in setup with their Intel E810 NIC with error messages saying that the driver cannot allocate enough MSI-X vectors on their 256 cpu-count system. It seems the ICE ethernet driver has an all or nothing approach to allocating MSI-X vectors and could request more MSI-X vectors than it finds available, which could lead to the driver failing to initialize and start. [Fix] The patch that fixes this allocates as many MSI-X vectors as it can to continue functionality by reducing the number of requested MSI-X vectors if it does not have enough to do full allocation. [Backport] In Jammy we do not carry patches for switchdev support in the driver so do not allocate the switchdev MSI-X vector for it. Also in Jammy use the older way of checking RDMA support by testing the RDMA bit is set as opposed to the newer ice_is_rdma_ena that the patch uses. [Test Plan] Install and startup Ice driver with an Intel 800 series NIC and check that we do not have the failure: Not enough device MSI-X vectors, requested = 260, available = 253 and check that everything works as expected. The backported patch for Jammy has been tested by the original user who submited the bug report with their high cpu count system and confirmed no errors. [Where problems could occur] There could be problems with the logic of reducing the MSI-X vector usage leading to more errors in the driver, but otherwise minimal regression potential as the code is mostly refactoring initial MSI-X setup. -- System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
** Tags removed: verification-needed-kinetic ** Tags added: verification-done-kinetic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: Fix Committed Status in linux source package in Kinetic: Fix Committed Bug description: SRU Justification: [Impact] There is a user reporting errors in setup with their Intel E810 NIC with error messages saying that the driver cannot allocate enough MSI-X vectors on their 256 cpu-count system. It seems the ICE ethernet driver has an all or nothing approach to allocating MSI-X vectors and could request more MSI-X vectors than it finds available, which could lead to the driver failing to initialize and start. [Fix] The patch that fixes this allocates as many MSI-X vectors as it can to continue functionality by reducing the number of requested MSI-X vectors if it does not have enough to do full allocation. [Backport] In Jammy we do not carry patches for switchdev support in the driver so do not allocate the switchdev MSI-X vector for it. Also in Jammy use the older way of checking RDMA support by testing the RDMA bit is set as opposed to the newer ice_is_rdma_ena that the patch uses. [Test Plan] Install and startup Ice driver with an Intel 800 series NIC and check that we do not have the failure: Not enough device MSI-X vectors, requested = 260, available = 253 and check that everything works as expected. The backported patch for Jammy has been tested by the original user who submited the bug report with their high cpu count system and confirmed no errors. [Where problems could occur] There could be problems with the logic of reducing the MSI-X vector usage leading to more errors in the driver, but otherwise minimal regression potential as the code is mostly refactoring initial MSI-X setup. -- System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
Done verification with linux/5.15.0-74.81, and got a PASS result, thanks ** Tags removed: verification-needed-jammy ** Tags added: verification-done-jammy -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: Fix Committed Status in linux source package in Kinetic: Fix Committed Bug description: SRU Justification: [Impact] There is a user reporting errors in setup with their Intel E810 NIC with error messages saying that the driver cannot allocate enough MSI-X vectors on their 256 cpu-count system. It seems the ICE ethernet driver has an all or nothing approach to allocating MSI-X vectors and could request more MSI-X vectors than it finds available, which could lead to the driver failing to initialize and start. [Fix] The patch that fixes this allocates as many MSI-X vectors as it can to continue functionality by reducing the number of requested MSI-X vectors if it does not have enough to do full allocation. [Backport] In Jammy we do not carry patches for switchdev support in the driver so do not allocate the switchdev MSI-X vector for it. Also in Jammy use the older way of checking RDMA support by testing the RDMA bit is set as opposed to the newer ice_is_rdma_ena that the patch uses. [Test Plan] Install and startup Ice driver with an Intel 800 series NIC and check that we do not have the failure: Not enough device MSI-X vectors, requested = 260, available = 253 and check that everything works as expected. The backported patch for Jammy has been tested by the original user who submited the bug report with their high cpu count system and confirmed no errors. [Where problems could occur] There could be problems with the logic of reducing the MSI-X vector usage leading to more errors in the driver, but otherwise minimal regression potential as the code is mostly refactoring initial MSI-X setup. -- System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
Thank you for confirmation, will check it again at that time (the beginning of June) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: In Progress Status in linux source package in Kinetic: In Progress Bug description: SRU Justification: [Impact] There is a user reporting errors in setup with their Intel E810 NIC with error messages saying that the driver cannot allocate enough MSI-X vectors on their 256 cpu-count system. It seems the ICE ethernet driver has an all or nothing approach to allocating MSI-X vectors and could request more MSI-X vectors than it finds available, which could lead to the driver failing to initialize and start. [Fix] The patch that fixes this allocates as many MSI-X vectors as it can to continue functionality by reducing the number of requested MSI-X vectors if it does not have enough to do full allocation. [Backport] In Jammy we do not carry patches for switchdev support in the driver so do not allocate the switchdev MSI-X vector for it. Also in Jammy use the older way of checking RDMA support by testing the RDMA bit is set as opposed to the newer ice_is_rdma_ena that the patch uses. [Test Plan] Install and startup Ice driver with an Intel 800 series NIC and check that we do not have the failure: Not enough device MSI-X vectors, requested = 260, available = 253 and check that everything works as expected. The backported patch for Jammy has been tested by the original user who submited the bug report with their high cpu count system and confirmed no errors. [Where problems could occur] There could be problems with the logic of reducing the MSI-X vector usage leading to more errors in the driver, but otherwise minimal regression potential as the code is mostly refactoring initial MSI-X setup. -- System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
@lukenow, do you have estimated time required to complete this ticket, it is under "in progress" currently, thanks -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: In Progress Status in linux source package in Kinetic: In Progress Bug description: SRU Justification: [Impact] There is a user reporting errors in setup with their Intel E810 NIC with error messages saying that the driver cannot allocate enough MSI-X vectors on their 256 cpu-count system. It seems the ICE ethernet driver has an all or nothing approach to allocating MSI-X vectors and could request more MSI-X vectors than it finds available, which could lead to the driver failing to initialize and start. [Fix] The patch that fixes this allocates as many MSI-X vectors as it can to continue functionality by reducing the number of requested MSI-X vectors if it does not have enough to do full allocation. [Backport] In Jammy we do not carry patches for switchdev support in the driver so do not allocate the switchdev MSI-X vector for it. Also in Jammy use the older way of checking RDMA support by testing the RDMA bit is set as opposed to the newer ice_is_rdma_ena that the patch uses. [Test Plan] Install and startup Ice driver with an Intel 800 series NIC and check that we do not have the failure: Not enough device MSI-X vectors, requested = 260, available = 253 and check that everything works as expected. The backported patch for Jammy has been tested by the original user who submited the bug report with their high cpu count system and confirmed no errors. [Where problems could occur] There could be problems with the logic of reducing the MSI-X vector usage leading to more errors in the driver, but otherwise minimal regression potential as the code is mostly refactoring initial MSI-X setup. -- System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
@lukenow, after installing the modules you provided, the issue is gone when you have the final kernel version included this, please update here, thanks -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: In Progress Bug description: System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
Hello, is there anyone who can help here ? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Confirmed Bug description: System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2012335] Re: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
It's not a system crash, but a kernel driver related issues for PCIe NIC ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2012335 Title: Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256) Status in linux package in Ubuntu: Confirmed Bug description: System Configuration OS: Ubuntu 22.04 LTS Kernel: 5.15.0-25-generic CPUs: 256 NIC: Intel E810 NIC with 512 MSIx vectors each function Errors Not enough device MSI-X vectors, requested = 260, available = 253 Findings (1) the current ice kernel driver (ice_main.c) will pre-allocate all required number of msix (even it's not enough for big core CPUs) (2) the commit https://github.com/torvalds/linux/commit/ce4626131112e1d0066a890371e14d8091323f99 has improved this logic, and it seems merged into kernel version from v6.1 So for supporting the new CPUs with more than 252 vCPUs, will Ubuntu kernel backport above patch to the current kernel (v5.15) ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012335/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
