Hi, Any suggestions? I tried with Centos 7 also and it has the same issue. Regards, Nitin
-----Original Message----- From: Nitin Katiyar Sent: Thursday, January 25, 2024 2:44 PM To: [email protected] Subject: Failed to enable SRIOV Hi, I am trying to enable SR-IOV on linux server (details below) and it throws following error: # echo '4' > /sys/class/net/enp1s0f0/device/sriov_numvfs -bash: echo: write error: Cannot allocate memory I went through some of articles on internet and it suggested some grub changes for pci but none of them helped. Following are the details of system and error System details: ------------------- Rocky Linux release 9.3 (Blue Onyx) model name : Intel(R) Xeon(R) E-2236 CPU @ 3.40GHz flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp vnmi md_clear flush_l1d arch_capabilities vmx flags : vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple shadow_vmcs pml ept_mode_based_exec ixgbe ver: 5.14.0-362.8.1.el9_3.x86_64 [root@server221 dpdk-stable-22.11.1]# uname -r 5.14.0-362.8.1.el9_3.x86_64 # for d in $(find /sys/kernel/iommu_groups/ -type l | sort -n -k5 -t/); do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done; IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3eca] (rev 07) IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07) IOMMU Group 1 01:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) IOMMU Group 1 01:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) IOMMU Group 2 00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911] IOMMU Group 3 00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10) IOMMU Group 4 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10) IOMMU Group 4 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10) IOMMU Group 5 00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 [8086:a368] (rev 10) IOMMU Group 5 00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 [8086:a369] (rev 10) IOMMU Group 6 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10) IOMMU Group 6 00:16.4 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller #2 [8086:a364] (rev 10) IOMMU Group 7 00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10) IOMMU Group 8 00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 [8086:a32c] (rev f0) IOMMU Group 9 00:1b.5 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #22 [8086:a32d] (rev f0) IOMMU Group 10 00:1b.6 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #23 [8086:a32e] (rev f0) IOMMU Group 11 00:1b.7 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #24 [8086:a32f] (rev f0) IOMMU Group 12 00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0) IOMMU Group 13 00:1e.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller [8086:a328] (rev 10) IOMMU Group 14 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a30a] (rev 10) IOMMU Group 14 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10) IOMMU Group 14 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10) IOMMU Group 15 02:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) IOMMU Group 16 03:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) IOMMU Group 17 04:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) IOMMU Group 18 05:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) IOMMU Group 19 06:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04) IOMMU Group 19 07:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41) Grub: intel_iommu=on iommu=pt pci=realloc pci_pt_e820_access=on [1089685.120029] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0 [1089685.120334] ixgbe 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link) [1089685.120417] ixgbe 0000:01:00.0: MAC: 2, PHY: 15, SFP+: 7, PBA No: 0210FF-0FF [1089685.120419] ixgbe 0000:01:00.0: 0c:c4:7a:8e:9e:70 [1089685.123311] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection [1089685.123945] libphy: ixgbe-mdio: probed [1089685.124909] ixgbe 0000:01:00.0 enp1s0f0: renamed from eth0 [1089685.193966] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0 [1089685.353711] ixgbe 0000:01:00.0 enp1s0f0: detected SFP+: 7 [1089685.816670] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX [1089685.816793] IPv6: ADDRCONF(NETDEV_CHANGE): enp1s0f0: link becomes ready [1089688.381885] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0 [1089688.382173] ixgbe 0000:01:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link) [1089688.382267] ixgbe 0000:01:00.1: MAC: 2, PHY: 15, SFP+: 65535, PBA No: 0210FF-0FF [1089688.382268] ixgbe 0000:01:00.1: 0c:c4:7a:8e:9e:71 [1089688.385318] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection [1089688.385422] libphy: ixgbe-mdio: probed [1089688.390327] ixgbe 0000:01:00.1 enp1s0f1: renamed from eth0 [1089688.458940] ixgbe 0000:01:00.1: registered PHC device on enp1s0f1 [1089688.566892] ixgbe 0000:01:00.1 enp1s0f1: detected SFP+: 65535 [1089711.433915] ixgbe 0000:01:00.0 enp1s0f0: SR-IOV enabled with 4 VFs [1089711.434093] ixgbe 0000:01:00.0: removed PHC on enp1s0f0 [1089711.491490] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0 [1089711.509777] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0 [1089711.616376] ixgbe 0000:01:00.0: can't enable 4 VFs (bus 02 out of range of [bus 01]) [1089711.616379] ixgbe 0000:01:00.0: Failed to enable PCI sriov: -12 [1089711.683690] ixgbe 0000:01:00.0 enp1s0f0: detected SFP+: 7 [1089712.108505] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX Grub = intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses [ 1097.055411] ixgbe 0000:01:00.0: removed PHC on enp1s0f0 [ 1097.113139] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0 [ 1097.131961] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0 [ 1097.237630] ixgbe 0000:01:00.0: not enough MMIO resources for SR-IOV [ 1097.237632] ixgbe 0000:01:00.0: Failed to enable PCI sriov: -12 [ 1097.301447] ixgbe 0000:01:00.0 enp1s0f0: detected SFP+: 7 [ 1097.576271] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX Grub: intel_iommu=on iommu=pt pci_pt_e820_access=on [ 62.686491] ixgbe 0000:01:00.0 enp1s0f0: SR-IOV enabled with 4 VFs [ 62.686644] ixgbe 0000:01:00.0: removed PHC on enp1s0f0 [ 62.744406] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0 [ 62.762909] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0 [ 62.869644] ixgbe 0000:01:00.0: not enough MMIO resources for SR-IOV [ 62.869646] ixgbe 0000:01:00.0: Failed to enable PCI sriov: -12 [ 62.933507] ixgbe 0000:01:00.0 enp1s0f0: detected SFP+: 7 [ 63.285637] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX [ 75.310635] block sda: the capability attribute has been deprecated. Grub: intel_iommu=on iommu=pt pci_pt_e820_access=on pci=realloc [ 67.969725] ixgbe 0000:01:00.0 enp1s0f0: SR-IOV enabled with 4 VFs [ 67.969891] ixgbe 0000:01:00.0: removed PHC on enp1s0f0 [ 68.026892] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0 [ 68.045699] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0 [ 68.149395] ixgbe 0000:01:00.0: can't enable 4 VFs (bus 02 out of range of [bus 01]) [ 68.149398] ixgbe 0000:01:00.0: Failed to enable PCI sriov: -12 [ 68.213069] ixgbe 0000:01:00.0 enp1s0f0: detected SFP+: 7 [ 68.669423] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX Any suggestions/help will be appreciated. Thanks, Nitin
