I was able to get things to work with rdma-core by completely deleting my DPDK build directory and building from scratch, not just clean.
The latest OFED (5.4) also works with kernel 5.4.0-80-generic, in case others are also having similar issues and cannot use rdma-core. On Tue, Aug 3, 2021 at 7:24 AM PATRICK KEROULAS < patrick.kerou...@radio-canada.ca> wrote: > Have you checked that: > - the missing symbols like `mlx5dv_dr_action_create_default_miss` are > defined in mlx5dv.h? > - there is no other mlx5dv.h from previous version installed somewhere? > > I recommend to keep the package versions provided for your distro and > downgrade dpdk if necessary. > > > On Mon, Aug 2, 2021 at 4:18 PM Gerry Wan <ger...@stanford.edu> wrote: > > > > I am running Ubuntu 20.04, and the mlx5dv.h is indeed in > `/usr/include/infiniband/`. > > > > The steps I take to build DPDK are > > $ meson --prefix=$DPDK_PATH build > > $ cd build > > $ sudo ninja install -> fails here > > $ sudo ldconfig > > > > How can I install the correct versions of rdma-core and libibverbs? > > > > On Mon, Aug 2, 2021 at 12:59 PM PATRICK KEROULAS < > patrick.kerou...@radio-canada.ca> wrote: > >> > >> I guess you're running Focal or Groovy release so rdma-core and > libibverbs > >> are >v28: https://launchpad.net/rdma-core > >> For this version, your missing symbols are defined: > >> > https://github.com/linux-rdma/rdma-core/blob/stable-v28/providers/mlx5/mlx5dv.h > >> I expect this header to be installed in `/usr/include/infiniband/`. > >> Please verify. > >> > >> Maybe something wrong with ninja, try meson or make. > >> > >> On Mon, Aug 2, 2021 at 2:35 PM Gerry Wan <ger...@stanford.edu> wrote: > >> > > >> > How did you install these packages? Are there extra environment > variables I need to set? > >> > After running the OFED uninstall script, > >> > > >> > apt-get install rdma-core > >> > apt-get install libibverbs-dev > >> > > >> > both work fine, but ninja install now fails on > >> > > >> > [1/1756] Linking target drivers/librte_common_mlx5.so.21.1. > >> > FAILED: drivers/librte_common_mlx5.so.21.1 > >> > cc -o drivers/librte_common_mlx5.so.21.1 'drivers/a715181@ > @rte_common_mlx5@sha/meson-generated_.._rte_common_mlx5.pmd.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_devx_cmds.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mp.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mr.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_malloc.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_pci.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_devx.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_nl.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_os.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_verbs.c.o' > 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o' > -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group > -Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl > -lnuma lib/librte_hash.so.21.1 lib/librte_eal.so.21.1 > lib/librte_kvargs.so.21.1 lib/librte_telemetry.so.21.1 > lib/librte_ring.so.21.1 lib/librte_rcu.so.21.1 lib/librte_pci.so.21.1 > drivers/librte_bus_pci.so.21.1 lib/librte_net.so.21.1 > lib/librte_mbuf.so.21.1 lib/librte_mempool.so.21.1 > -Wl,--version-script=/home/gerryw/dpdk/dpdk-21.02/drivers/common/mlx5/version.map > /usr/lib/x86_64-linux-gnu/libmlx5.so > /usr/lib/x86_64-linux-gnu/libibverbs.so -Wl,--end-group > '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/' > -Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/lib > -Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/drivers > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dv_create_flow_action_aso': > >> > mlx5_glue.c:(.text+0x21f): undefined reference to > `mlx5dv_dr_action_create_aso' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dr_action_create_dest_array': > >> > mlx5_glue.c:(.text+0x23b): undefined reference to > `mlx5dv_dr_action_create_dest_array' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dr_create_flow_action_sampler': > >> > mlx5_glue.c:(.text+0x25b): undefined reference to > `mlx5dv_dr_action_create_flow_sampler' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dv_alloc_pp': > >> > mlx5_glue.c:(.text+0x28b): undefined reference to `mlx5dv_pp_alloc' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dr_create_flow_action_default_miss': > >> > mlx5_glue.c:(.text+0x4bb): undefined reference to > `mlx5dv_dr_action_create_default_miss' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dv_free_pp': > >> > mlx5_glue.c:(.text+0x277): undefined reference to `mlx5dv_pp_free' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dr_reclaim_domain_memory': > >> > mlx5_glue.c:(.text+0x2a7): undefined reference to > `mlx5dv_dr_domain_set_reclaim_device_memory' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_devx_port_query': > >> > mlx5_glue.c:(.text+0x325): undefined reference to > `mlx5dv_query_devx_port' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dv_create_flow_action_dest_devx_tir': > >> > mlx5_glue.c:(.text+0x545): undefined reference to > `mlx5dv_dr_action_create_dest_devx_tir' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dr_create_flow_action_pop_vlan': > >> > mlx5_glue.c:(.text+0x655): undefined reference to > `mlx5dv_dr_action_create_pop_vlan' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dr_create_flow_action_push_vlan': > >> > mlx5_glue.c:(.text+0x665): undefined reference to > `mlx5dv_dr_action_create_push_vlan' > >> > /usr/bin/ld: > >> > drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: > in function `mlx5_glue_dr_create_flow_action_dest_port': > >> > mlx5_glue.c:(.text+0x685): undefined reference to > `mlx5dv_dr_action_create_dest_ib_port' > >> > collect2: error: ld returned 1 exit status > >> > [2/1756] Compiling C object 'drivers/a715181@ > @tmp_rte_mempool_octeontx@sta/mempool_octeontx_octeontx_fpavf.c.o'. > >> > ninja: build stopped: subcommand failed. > >> > > >> > > >> > On Mon, Aug 2, 2021 at 6:24 AM PATRICK KEROULAS < > patrick.kerou...@radio-canada.ca> wrote: > >> >> > >> >> > >> >> Hello Gerry, > >> >> I went through this a while ago and had to properly remove > everything related > >> >> to OFED with the uninstaller. Then, install rdma-core and > libibverbs-dev which > >> >> includes infiniband/mlx5dv.h IIRC. > >> >> Regards, > >> >> Patrick > >> >> > >> >> On Sun, Aug 1, 2021 at 8:23 PM Gerry Wan <ger...@stanford.edu> > wrote: > >> >>> > >> >>> I recently tried to reinstall MLX OFED 5.2-2.2.0.0 on my bare metal > machine > >> >>> Ubuntu 20.04 (previously using the command ./mlnxofedinstall --dpdk > >> >>> --upstream-libs --with-mft --with-kernel-mft). However, it now > fails with > >> >>> the message > >> >>> > >> >>> Checking SW Requirements... > >> >>> Removing old packages... > >> >>> Uninstalling the previous version of MLNX_OFED_LINUX > >> >>> Installing new packages Installing mft-4.16.1... > >> >>> Installing ofed-scripts-5.2... > >> >>> Installing mstflint-4.15.0... > >> >>> Installing mlnx-ofed-kernel-utils-5.2... > >> >>> Installing mlnx-ofed-kernel-dkms-5.2... > >> >>> Failed to install mlnx-ofed-kernel-dkms DEB > >> >>> Collecting debug info... > >> >>> See > /tmp/MLNX_OFED_LINUX.9221.logs/mlnx-ofed-kernel-dkms.debinstall.log > >> >>> > >> >>> It appears that my kernel version got upgraded to 5.4.0-80-generic > which I > >> >>> think may be the issue. I've also tried to install using > >> >>> --add-kernel-support --kernel 5.4.0-80-generic, with no luck. > >> >>> > >> >>> The DPDK website says that rdma-core is now recommended over OFED, > but the > >> >>> documentation for how to use it isn't very clear to me. I've tried > to > >> >>> install it via > >> >>> > >> >>> apt-get install rdma-core > >> >>> > >> >>> as well as running the quick start script listed here: > >> >>> https://github.com/linux-rdma/rdma-core, but DPDK installation > still fails > >> >>> with > >> >>> > >> >>> ../drivers/common/mlx5/linux/mlx5_glue.h:15:10: fatal error: > >> >>> infiniband/mlx5dv.h: No such file or directory > >> >>> 15 | #include <infiniband/mlx5dv.h> > >> >>> > >> >>> Any advice on getting the mlx5 drivers to work again would be > helpful! > >> > >
