Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1
gt; hca_id: mlx5_1 >> >> transport: InfiniBand (0) >> >> fw_ver: 14.32.1010 >> >> node_guid: 0c42:a103:007e:a9b9 >> >> sys_image_guid: 0c42:a103:007e:a9b8 >> >> vendor_id: 0x02c9 >> >> vendor_part_id: 4117 >> >> hw_ver: 0x0 >> >> board_id: MT_2420110034 >> >> phys_port_cnt: 1 >> >> port: 1 >> >> state: PORT_ACTIVE (4) >> >> max_mtu: 4096 (5) >> >> active_mtu: 1024 (3) >> >> sm_lid: 0 >> >> port_lid: 0 >> >> port_lmc: 0x00 >> >> link_layer: Ethernet >> >> On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber wrote: >> >>> Hi, >>> >>> Based on your output, the ConnectX-4LX device is configured in LAG mode >>> managed via the kernel bonding scripts. In this mode, both physical >>> functions share a single port (mlx5_bond_0). You should only probe the >>> first PCI BDF - 01:00.0, not the 2nd one. >>> >>> By the way, the --dpdk installation flag should not be necessary, it is >>> an old flag keps for Mellanox OFED builds lower than 5.x. >>> >>> Regards, >>> Erez >>> >>> On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532ya...@gmail.com> wrote: >>> >>>> I built the current version of DPDK directly from dpdk.org after I >>>> installed the current OFED Mellanox driver set: >>>> >>>> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso >>>> >>>> with ./install --dpdk >>>> >>>> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which >>>> is Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034 >>>> board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI >>>> device :01:00.1, are kernel drivers loaded?" >>>> >>>> EAL: Detected CPU lcores: 16 >>>> EAL: Detected NUMA nodes: 1 >>>> EAL: Detected static linkage of DPDK >>>> EAL: Selected IOVA mode 'PA' >>>> EAL: No free 2048 kB hugepages reported on node 0 >>>> EAL: VFIO support initialized >>>> EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: :01:00.1 >>>> (socket 0) >>>> mlx5_common: No Verbs device matches PCI device :01:00.1, are >>>> kernel drivers loaded? >>>> mlx5_common: Verbs device not found: 01:00.1 >>>> mlx5_common: Failed to initialize device context. >>>> EAL: Requested device :01:00.1 cannot be used >>>> EAL: Bus (pci) probe failed. >>>> >>>> As far as I can see all the kernel modules are loaded: >>>> >>>> lsmod | egrep "(ib|mlx)" | sort >>>> ib_cm 53248 2 rdma_cm,ib_ipoib >>>> ib_core 368640 8 >>>> rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm >>>> ib_ipoib 135168 0 >>>> ib_umad24576 0 >>>> ib_uverbs 139264 2 rdma_ucm,mlx5_ib >>>> libahci36864 1 ahci >>>> libcrc32c 16384 2 btrfs,raid456 >>>> mlx5_core1634304 1 mlx5_ib >>>> mlx5_ib 397312 0 >>>> mlx_compat 69632 11 >>>> rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core >>>> mlxdevm 172032 1 mlx5_core >>>> mlxfw 32768 1 mlx5_core >>>> pci_hyperv_intf16384 1 mlx5_core >>>> psample20480 1 mlx5_core >>>> tls94208 2 bonding,mlx5_core >>>> >>>> root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v >>>> MST modules: >>>> >>>> MST PCI module is not loaded >>>> MST PCI configuration module loaded >>>> PCI devices: >>>> >>>> DEVICE_TYPE MST PCI RDMA >>>> NET NUMA >>>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.101:00.1 >>>> mlx5_bond_0 net-bond0 -1 >>>> >>>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0 >>>> mlx5_bond_0 net-bond0 -1 >>>> >>>
Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1
I assume your tree assumes there's a ConnectX-3 device installed, while the kernel driver doesnt support it for quite a while I would suggest re-compiling while excluding mlx4 PMD support. Thanks, Erez On Tue, 5 Apr 2022 at 23:54, fwefew 4t4tg <7532ya...@gmail.com> wrote: > Mr. Ferber, much appreciated. I knew this metal box came to me with two > mellanox NICs bonded. I used their util to unbond it, but alas it did not > do it all the way. The /etc/network/interfaces was bad. I fixed the config > and rebooted. > > Voila. Success. ibv_devinfo now shows two devices. And the DPDK app runs. > > Your point about bonding nudged me to double check. > > *Now, on a second machine where I believe I have done everything I did on > the first machine*, running the DPDK application continues to look for > the wrong driver: > > EAL: Detected CPU lcores: 16 > EAL: Detected NUMA nodes: 1 > EAL: Detected shared linkage of DPDK > EAL: libmlx4.so.1: cannot open shared object file: No such file or > directory > EAL: FATAL: Cannot init plugins > EAL: Cannot init plugins > > Somehow I fixed this on the first machine, but I cannot duplicate success > on the second machine. > > Both machines report two devices; both machines show similar ifconifgs > > device node GUID > > -- > > mlx5_0 0c42a103007ea9b8 > > mlx5_1 0c42a103007ea9b9 > > > device node GUID > > -- > > mlx5_0 0c42a103007ea3ec > > mlx5_1 0c42a103007ea3ed > > > root@server:~/Dev/reinvent/scripts# ibv_devinfo > > hca_id: mlx5_0 > > transport: InfiniBand (0) > > fw_ver: 14.32.1010 > > node_guid: 0c42:a103:007e:a3ec > > sys_image_guid: 0c42:a103:007e:a3ec > > vendor_id: 0x02c9 > > vendor_part_id: 4117 > > hw_ver: 0x0 > > board_id: MT_2420110034 > > phys_port_cnt: 1 > > port: 1 > > state: PORT_ACTIVE (4) > > max_mtu: 4096 (5) > > active_mtu: 1024 (3) > > sm_lid: 0 > > port_lid: 0 > > port_lmc: 0x00 > > link_layer: Ethernet > > > hca_id: mlx5_1 > > transport: InfiniBand (0) > > fw_ver: 14.32.1010 > > node_guid: 0c42:a103:007e:a3ed > > sys_image_guid: 0c42:a103:007e:a3ec > > vendor_id: 0x02c9 > > vendor_part_id: 4117 > > hw_ver: 0x0 > > board_id: MT_2420110034 > > phys_port_cnt: 1 > > port: 1 > > state: PORT_ACTIVE (4) > > max_mtu: 4096 (5) > > active_mtu: 1024 (3) > > sm_lid: 0 > > port_lid: 0 > > port_lmc: 0x00 > > link_layer: Ethernet > > > > > > ibv_devinfo > > hca_id: mlx5_0 > > transport: InfiniBand (0) > > fw_ver: 14.32.1010 > > node_guid: 0c42:a103:007e:a9b8 > > sys_image_guid: 0c42:a103:007e:a9b8 > > vendor_id: 0x02c9 > > vendor_part_id: 4117 > > hw_ver: 0x0 > > board_id: MT_2420110034 > > phys_port_cnt: 1 > > port: 1 > > state: PORT_ACTIVE (4) > > max_mtu: 4096 (5) > > active_mtu: 1024 (3) > > sm_lid: 0 > > port_lid: 0 > > port_lmc: 0x00 > > link_layer: Ethernet > > > hca_id: mlx5_1 > > transport: InfiniBand (0) > > fw_ver: 14.32.1010 > > node_guid: 0c42:a103:007e:a9b9 > > sys_image_guid: 0c42:a103:007e:a9b8 > > vendor_id: 0x02c9 > > vendor_part_id: 4117 > > hw_ver: 0x0 > > board_id: MT_2420110034 > > phys_port_cnt: 1 > > port: 1 > > state: PORT_ACTIVE (4) > > max_mtu: 4096 (5) > > active_mtu: 1024 (3) > > sm_lid: 0 > > port_lid: 0 > > port_lmc: 0x00 > > link_layer: Ethernet > > On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber wrote: > >> Hi, >> >> Based on your output, the ConnectX-4LX device is configured in LAG mode >> managed via the kernel bonding scripts. In this mode, both physical >> functions share a single port (mlx5_bond_0). You should only probe the >> first PCI BDF - 01:00.0, not the 2nd one. >> >> By the way, the --dpdk installation flag should not be necessary, it is >> an old flag keps for Mellanox OFED builds lower than 5.x. >> >> Regards, >> Erez >> >> On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532ya...@gmail.com> wrote: >> >>> I built the current version of DPDK directly from dpdk.org after I >>> installed the current OFED Mellanox driver set: >>> >>> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso >>> >>> with ./install --dpdk >>> >>> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which >>> is Ethernet
Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1
Mr. Ferber, much appreciated. I knew this metal box came to me with two mellanox NICs bonded. I used their util to unbond it, but alas it did not do it all the way. The /etc/network/interfaces was bad. I fixed the config and rebooted. Voila. Success. ibv_devinfo now shows two devices. And the DPDK app runs. Your point about bonding nudged me to double check. *Now, on a second machine where I believe I have done everything I did on the first machine*, running the DPDK application continues to look for the wrong driver: EAL: Detected CPU lcores: 16 EAL: Detected NUMA nodes: 1 EAL: Detected shared linkage of DPDK EAL: libmlx4.so.1: cannot open shared object file: No such file or directory EAL: FATAL: Cannot init plugins EAL: Cannot init plugins Somehow I fixed this on the first machine, but I cannot duplicate success on the second machine. Both machines report two devices; both machines show similar ifconifgs device node GUID -- mlx5_0 0c42a103007ea9b8 mlx5_1 0c42a103007ea9b9 device node GUID -- mlx5_0 0c42a103007ea3ec mlx5_1 0c42a103007ea3ed root@server:~/Dev/reinvent/scripts# ibv_devinfo hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 14.32.1010 node_guid: 0c42:a103:007e:a3ec sys_image_guid: 0c42:a103:007e:a3ec vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: MT_2420110034 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 14.32.1010 node_guid: 0c42:a103:007e:a3ed sys_image_guid: 0c42:a103:007e:a3ec vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: MT_2420110034 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet ibv_devinfo hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 14.32.1010 node_guid: 0c42:a103:007e:a9b8 sys_image_guid: 0c42:a103:007e:a9b8 vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: MT_2420110034 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 14.32.1010 node_guid: 0c42:a103:007e:a9b9 sys_image_guid: 0c42:a103:007e:a9b8 vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: MT_2420110034 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber wrote: > Hi, > > Based on your output, the ConnectX-4LX device is configured in LAG mode > managed via the kernel bonding scripts. In this mode, both physical > functions share a single port (mlx5_bond_0). You should only probe the > first PCI BDF - 01:00.0, not the 2nd one. > > By the way, the --dpdk installation flag should not be necessary, it is an > old flag keps for Mellanox OFED builds lower than 5.x. > > Regards, > Erez > > On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532ya...@gmail.com> wrote: > >> I built the current version of DPDK directly from dpdk.org after I >> installed the current OFED Mellanox driver set: >> >> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso >> >> with ./install --dpdk >> >> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which >> is Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034 >> board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI >> device :01:00.1, are kernel drivers loaded?" >> >> EAL: Detected CPU lcores: 16 >> EAL: Detected NUMA nodes: 1 >> EAL: Detected static linkage of DPDK >> EAL: Selected IOVA mode 'PA' >> EAL: No free 2048 kB hugepages reported on node 0 >> EAL: VFIO support initialized >> EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: :01:00.1 (socket >> 0) >> mlx5_common: No Verbs device matches PCI device :01:00.1, are kernel >> drivers loaded? >> mlx5_common: Verbs device not found: 01:00.1 >> mlx5_common: Failed to initialize device context. >> EAL: Requested device :01:00.1 cannot be used >> EAL: Bus (pci) probe failed. >> >> As far as I can see all the kernel modules are loaded: >> >> lsmod | egrep "(ib|mlx)" | sort >> ib_cm 53248 2 rdma_cm,ib_ipoib >> ib_core 368640 8 >> rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm >> ib_ipoib 135168 0 >> ib_umad24576 0 &
Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1
Hi, Based on your output, the ConnectX-4LX device is configured in LAG mode managed via the kernel bonding scripts. In this mode, both physical functions share a single port (mlx5_bond_0). You should only probe the first PCI BDF - 01:00.0, not the 2nd one. By the way, the --dpdk installation flag should not be necessary, it is an old flag keps for Mellanox OFED builds lower than 5.x. Regards, Erez On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532ya...@gmail.com> wrote: > I built the current version of DPDK directly from dpdk.org after I > installed the current OFED Mellanox driver set: > > * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso > > with ./install --dpdk > > I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which is > Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034 > board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI > device :01:00.1, are kernel drivers loaded?" > > EAL: Detected CPU lcores: 16 > EAL: Detected NUMA nodes: 1 > EAL: Detected static linkage of DPDK > EAL: Selected IOVA mode 'PA' > EAL: No free 2048 kB hugepages reported on node 0 > EAL: VFIO support initialized > EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket 0) > mlx5_common: No Verbs device matches PCI device :01:00.1, are kernel > drivers loaded? > mlx5_common: Verbs device not found: 01:00.1 > mlx5_common: Failed to initialize device context. > EAL: Requested device :01:00.1 cannot be used > EAL: Bus (pci) probe failed. > > As far as I can see all the kernel modules are loaded: > > lsmod | egrep "(ib|mlx)" | sort > ib_cm 53248 2 rdma_cm,ib_ipoib > ib_core 368640 8 > rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm > ib_ipoib 135168 0 > ib_umad24576 0 > ib_uverbs 139264 2 rdma_ucm,mlx5_ib > libahci36864 1 ahci > libcrc32c 16384 2 btrfs,raid456 > mlx5_core1634304 1 mlx5_ib > mlx5_ib 397312 0 > mlx_compat 69632 11 > rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core > mlxdevm 172032 1 mlx5_core > mlxfw 32768 1 mlx5_core > pci_hyperv_intf16384 1 mlx5_core > psample20480 1 mlx5_core > tls94208 2 bonding,mlx5_core > > root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v > MST modules: > > MST PCI module is not loaded > MST PCI configuration module loaded > PCI devices: > > DEVICE_TYPE MST PCI RDMA > NET NUMA > ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.101:00.1 > mlx5_bond_0 net-bond0 -1 > > ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0 > mlx5_bond_0 net-bond0 -1 >
mlx5_common: No Verbs device matches PCI device 0000:01:00.1
I built the current version of DPDK directly from dpdk.org after I installed the current OFED Mellanox driver set: * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso with ./install --dpdk I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which is Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034 board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI device :01:00.1, are kernel drivers loaded?" EAL: Detected CPU lcores: 16 EAL: Detected NUMA nodes: 1 EAL: Detected static linkage of DPDK EAL: Selected IOVA mode 'PA' EAL: No free 2048 kB hugepages reported on node 0 EAL: VFIO support initialized EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket 0) mlx5_common: No Verbs device matches PCI device 0000:01:00.1, are kernel drivers loaded? mlx5_common: Verbs device not found: 01:00.1 mlx5_common: Failed to initialize device context. EAL: Requested device :01:00.1 cannot be used EAL: Bus (pci) probe failed. As far as I can see all the kernel modules are loaded: lsmod | egrep "(ib|mlx)" | sort ib_cm 53248 2 rdma_cm,ib_ipoib ib_core 368640 8 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm ib_ipoib 135168 0 ib_umad24576 0 ib_uverbs 139264 2 rdma_ucm,mlx5_ib libahci36864 1 ahci libcrc32c 16384 2 btrfs,raid456 mlx5_core1634304 1 mlx5_ib mlx5_ib 397312 0 mlx_compat 69632 11 rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core mlxdevm 172032 1 mlx5_core mlxfw 32768 1 mlx5_core pci_hyperv_intf16384 1 mlx5_core psample20480 1 mlx5_core tls94208 2 bonding,mlx5_core root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v MST modules: MST PCI module is not loaded MST PCI configuration module loaded PCI devices: DEVICE_TYPE MST PCI RDMA NET NUMA ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.101:00.1 mlx5_bond_0 net-bond0 -1 ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0 mlx5_bond_0 net-bond0 -1
