Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1
On Sun, 2019-04-07 at 11:31 +0200, Thomas Monjalon wrote: > 26/03/2019 19:05, Jason Gunthorpe: > > On Tue, Mar 26, 2019 at 02:51:25PM +0100, Thomas Monjalon wrote: > > > 26/03/2019 10:21, Christian Ehrhardt: > > > > > > rdma-core configures modprobe for ib/rdma modules, installs > > > > > > some > > > > > > helper daemons and installs some udev rules. It certainly > > > > > > makes > > > > > > administration of the server an easier task, but it isn't > > > > > > *required* > > > > > > when configuring a Debian server for rdma and user-space > > > > > > ibverbs. > > > > > > > > > > I didn't think it is required for mlx5 as the ethernet driver > > > > > autoloads the rdma modules itself.. > > > > > > > > Below is the log that I got from Mellanox (thanks Alaa and > > > > Thomas. > > > > To me the messages related to ib_uverbs seem interesting. > > > > > > > > @Jason - this is triggered by DPDK. In regard to the > > > > autoloading - did > > > > you refer to DPDKs net_mlx5 or the kernels net_mlx5 in e.g. > > > > mlx5_core.ko? > > > > I only found [1] which does not seem right, can you point at > > > > the > > > > autoloading feature (a commit maybe) that you meant (keep in > > > > mind this > > > > is DPDK 17.11.5)? > > > > [1]: > > > > https://doc.dpdk.org/guides/nics/mlx5.html#compilation-options > > > > > > > > > > There is no autoloading of kernel drivers when using DPDK. > > > > It looks like it is only done if the mlx5 NIC driver detects some > > kind > > of RDMA mode.. > > > > > > @Allaa/Thomas - due to the messages in the log, could you try > > > > if > > > > instead of installing rdma-core just doing the modprobe would > > > > be > > > > enough to resolve the problem? > > > > $ modprobe ib_uverbs # instead of the apt install > > > > > > Yes, manual modprobe partially replaces rdma-core packages. > > > > > > These are the files we identified as required from the rdma-core > > > packages: > > > /etc/modprobe.d/mlx4.conf > > > /etc/rdma/modules/rdma.conf > > > /lib/udev/rules.d/90-rdma-hw-modules.rules > > > /lib/systemd/system/rdma-hw.target > > > > > > The first file (mlx4.conf) is a template helping to get the right > > > config > > > by commenting out a line. > > > The others files are for autoloading of the kernel drivers. > > > > > > I think you should include the rdma-core package when using DPDK. > > > It can be a dependency of the DPDK package, or a dependency of > > > libibverbs. > > > Does it make sense to install libibverbs without rdma-core > > > configuration? > > > > Many rdma drivers do properly autoload and don't need the udev > > magic. > > > > I have a long term plan to avoid needing this, but nobody is > > working > > on it right now. > > So I guess the recommendation is to make DPDK depends on rdma-core > package. > Luca/Christian, is it possible to do this change in your packaging > please? Yes if the libraries can be used without the rdma-core in the general case then we can add a dependency via the mlx pmd. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1
26/03/2019 19:05, Jason Gunthorpe: > On Tue, Mar 26, 2019 at 02:51:25PM +0100, Thomas Monjalon wrote: > > 26/03/2019 10:21, Christian Ehrhardt: > > > > > > > > > > rdma-core configures modprobe for ib/rdma modules, installs some > > > > > helper daemons and installs some udev rules. It certainly makes > > > > > administration of the server an easier task, but it isn't *required* > > > > > when configuring a Debian server for rdma and user-space ibverbs. > > > > > > > > I didn't think it is required for mlx5 as the ethernet driver > > > > autoloads the rdma modules itself.. > > > > > > Below is the log that I got from Mellanox (thanks Alaa and Thomas. > > > To me the messages related to ib_uverbs seem interesting. > > > > > > @Jason - this is triggered by DPDK. In regard to the autoloading - did > > > you refer to DPDKs net_mlx5 or the kernels net_mlx5 in e.g. > > > mlx5_core.ko? > > > I only found [1] which does not seem right, can you point at the > > > autoloading feature (a commit maybe) that you meant (keep in mind this > > > is DPDK 17.11.5)? > > > [1]: https://doc.dpdk.org/guides/nics/mlx5.html#compilation-options > > > > There is no autoloading of kernel drivers when using DPDK. > > It looks like it is only done if the mlx5 NIC driver detects some kind > of RDMA mode.. > > > > @Allaa/Thomas - due to the messages in the log, could you try if > > > instead of installing rdma-core just doing the modprobe would be > > > enough to resolve the problem? > > > $ modprobe ib_uverbs # instead of the apt install > > > > Yes, manual modprobe partially replaces rdma-core packages. > > > > These are the files we identified as required from the rdma-core packages: > > /etc/modprobe.d/mlx4.conf > > /etc/rdma/modules/rdma.conf > > /lib/udev/rules.d/90-rdma-hw-modules.rules > > /lib/systemd/system/rdma-hw.target > > > > The first file (mlx4.conf) is a template helping to get the right config > > by commenting out a line. > > The others files are for autoloading of the kernel drivers. > > > > I think you should include the rdma-core package when using DPDK. > > It can be a dependency of the DPDK package, or a dependency of libibverbs. > > Does it make sense to install libibverbs without rdma-core configuration? > > Many rdma drivers do properly autoload and don't need the udev magic. > > I have a long term plan to avoid needing this, but nobody is working > on it right now. So I guess the recommendation is to make DPDK depends on rdma-core package. Luca/Christian, is it possible to do this change in your packaging please?
Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1
On Tue, Mar 26, 2019 at 02:51:25PM +0100, Thomas Monjalon wrote: > 26/03/2019 10:21, Christian Ehrhardt: > > > > > > > > rdma-core configures modprobe for ib/rdma modules, installs some > > > > helper daemons and installs some udev rules. It certainly makes > > > > administration of the server an easier task, but it isn't *required* > > > > when configuring a Debian server for rdma and user-space ibverbs. > > > > > > I didn't think it is required for mlx5 as the ethernet driver > > > autoloads the rdma modules itself.. > > > > Below is the log that I got from Mellanox (thanks Alaa and Thomas. > > To me the messages related to ib_uverbs seem interesting. > > > > @Jason - this is triggered by DPDK. In regard to the autoloading - did > > you refer to DPDKs net_mlx5 or the kernels net_mlx5 in e.g. > > mlx5_core.ko? > > I only found [1] which does not seem right, can you point at the > > autoloading feature (a commit maybe) that you meant (keep in mind this > > is DPDK 17.11.5)? > > [1]: https://doc.dpdk.org/guides/nics/mlx5.html#compilation-options > > There is no autoloading of kernel drivers when using DPDK. It looks like it is only done if the mlx5 NIC driver detects some kind of RDMA mode.. > > @Allaa/Thomas - due to the messages in the log, could you try if > > instead of installing rdma-core just doing the modprobe would be > > enough to resolve the problem? > > $ modprobe ib_uverbs # instead of the apt install > > Yes, manual modprobe partially replaces rdma-core packages. > > These are the files we identified as required from the rdma-core packages: > /etc/modprobe.d/mlx4.conf > /etc/rdma/modules/rdma.conf > /lib/udev/rules.d/90-rdma-hw-modules.rules > /lib/systemd/system/rdma-hw.target > > The first file (mlx4.conf) is a template helping to get the right config > by commenting out a line. > The others files are for autoloading of the kernel drivers. > > I think you should include the rdma-core package when using DPDK. > It can be a dependency of the DPDK package, or a dependency of libibverbs. > Does it make sense to install libibverbs without rdma-core configuration? Many rdma drivers do properly autoload and don't need the udev magic. I have a long term plan to avoid needing this, but nobody is working on it right now. Jason
Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1
26/03/2019 10:21, Christian Ehrhardt: > > > > > > rdma-core configures modprobe for ib/rdma modules, installs some > > > helper daemons and installs some udev rules. It certainly makes > > > administration of the server an easier task, but it isn't *required* > > > when configuring a Debian server for rdma and user-space ibverbs. > > > > I didn't think it is required for mlx5 as the ethernet driver > > autoloads the rdma modules itself.. > > Below is the log that I got from Mellanox (thanks Alaa and Thomas. > To me the messages related to ib_uverbs seem interesting. > > @Jason - this is triggered by DPDK. In regard to the autoloading - did > you refer to DPDKs net_mlx5 or the kernels net_mlx5 in e.g. > mlx5_core.ko? > I only found [1] which does not seem right, can you point at the > autoloading feature (a commit maybe) that you meant (keep in mind this > is DPDK 17.11.5)? > [1]: https://doc.dpdk.org/guides/nics/mlx5.html#compilation-options There is no autoloading of kernel drivers when using DPDK. > @Allaa/Thomas - due to the messages in the log, could you try if > instead of installing rdma-core just doing the modprobe would be > enough to resolve the problem? > $ modprobe ib_uverbs # instead of the apt install Yes, manual modprobe partially replaces rdma-core packages. These are the files we identified as required from the rdma-core packages: /etc/modprobe.d/mlx4.conf /etc/rdma/modules/rdma.conf /lib/udev/rules.d/90-rdma-hw-modules.rules /lib/systemd/system/rdma-hw.target The first file (mlx4.conf) is a template helping to get the right config by commenting out a line. The others files are for autoloading of the kernel drivers. I think you should include the rdma-core package when using DPDK. It can be a dependency of the DPDK package, or a dependency of libibverbs. Does it make sense to install libibverbs without rdma-core configuration?
Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1
> > > > rdma-core configures modprobe for ib/rdma modules, installs some > > helper daemons and installs some udev rules. It certainly makes > > administration of the server an easier task, but it isn't *required* > > when configuring a Debian server for rdma and user-space ibverbs. > > I didn't think it is required for mlx5 as the ethernet driver > autoloads the rdma modules itself.. Below is the log that I got from Mellanox (thanks Alaa and Thomas. To me the messages related to ib_uverbs seem interesting. @Jason - this is triggered by DPDK. In regard to the autoloading - did you refer to DPDKs net_mlx5 or the kernels net_mlx5 in e.g. mlx5_core.ko? I only found [1] which does not seem right, can you point at the autoloading feature (a commit maybe) that you meant (keep in mind this is DPDK 17.11.5)? @Allaa/Thomas - due to the messages in the log, could you try if instead of installing rdma-core just doing the modprobe would be enough to resolve the problem? $ modprobe ib_uverbs # instead of the apt install [1]: https://doc.dpdk.org/guides/nics/mlx5.html#compilation-options FYI - The test log from Mellanox showing that they need rdma-core to run properly: 1 - Running testpmd with installing any thing root@qa-l-vrt-111-124:~# testpmd -c 0xF -n 4 -w :00:06.0 -w :00:07.0 -- --txd=256 --rxd=256 --rxq=1 --txq=1 --nb-cores=3 --i EAL: Detected 4 lcore(s) EAL: Probing VFIO support... EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: PCI device :00:06.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 15b3:1016 net_mlx5 net_mlx5: cannot list devices, is ib_uverbs loaded? EAL: Requested device :00:06.0 cannot be used EAL: PCI device :00:07.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 15b3:1016 net_mlx5 net_mlx5: cannot list devices, is ib_uverbs loaded? EAL: Requested device :00:07.0 cannot be used EAL: No probed ethernet devices Interactive-mode selected USER1: create a new mbuf pool : n=171456, size=2176, socket=0 Done 2- Installing rdma-core root@qa-l-vrt-111-124:~# apt-get install rdma-core Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: rdma-core 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 56.8 kB of archives. After this operation, 195 kB of additional disk space will be used. Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 rdma-core amd64 17.1-1ubuntu0.1 [56.8 kB] Fetched 56.8 kB in 1s (79.4 kB/s) Selecting previously unselected package rdma-core. (Reading database ... 91266 files and directories currently installed.) Preparing to unpack .../rdma-core_17.1-1ubuntu0.1_amd64.deb ... Unpacking rdma-core (17.1-1ubuntu0.1) ... Processing triggers for ureadahead (0.100.0-20) ... Processing triggers for systemd (237-3ubuntu10.15) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... Setting up rdma-core (17.1-1ubuntu0.1) ... rdma-hw.target is a disabled or a static unit, not starting it. rdma-ndd.service is a disabled or a static unit, not starting it. Processing triggers for ureadahead (0.100.0-20) ... Processing triggers for systemd (237-3ubuntu10.15) ... 3- Running testpmd again after installing rdma-core root@qa-l-vrt-111-124:~# testpmd -c 0xF -n 4 -w :00:06.0 -w :00:07.0 -- --txd=256 --rxd=256 --rxq=1 --txq=1 --nb-cores=3 --i EAL: Detected 4 lcore(s) EAL: Probing VFIO support... EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: PCI device :00:06.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 15b3:1016 net_mlx5 EAL: PCI device :00:07.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 15b3:1016 net_mlx5 Interactive-mode selected USER1: create a new mbuf pool : n=171456, size=2176, socket=0 Configuring Port 0 (socket 0) Port 0: 26:D4:B6:45:70:D7 Configuring Port 1 (socket 0) Port 1: BA:52:9F:30:6E:BF Checking link statuses... Done
Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1
On Thu, Mar 21, 2019 at 01:26:04PM -0500, Brian Smith wrote: > On Thu, Mar 21, 2019 at 3:51 AM Christian Ehrhardt > wrote: > > > > > ibverbs is provided by rdma-core project. > > > Is it a separate package in your case? > > > > No it is part of the same src package. > > > > > Yes, you need libmlx4, libmlx5 and libibverbs. > > > It is described in the doc: > > > http://doc.dpdk.org/guides/nics/mlx5.html#prerequisites > > > > Well that is the problem, we fulfill that prereq already. > > libmlx4, libmlx5 and libibverbs are installed by dependencies already > > and thought all the time that would be good enough. > > > > But we got told that we also need the package rdma-core [1] itself to > > be installed to be useful. > > That was a discussion with the subject "Please help us to verify DPDK > > 17.11.5 for Ubuntu 18.04/18.10" with Mellanox test people. > > Fortunately I keep you on CC on those - so maybe you can find it > > somewhere in your inbox and contact the other Mellanox people on that > > thread. > > After that you might help to translate their request into our terms please? > > > > The question is: > > a) what exactly is (binary package) rdma-core [1] needed for in that > > scenario > > b) if (a) would be generally true we wondered why it had to be a dep > > of the DPDK packages and not of e.g. libibverbs1 itself > > That is why we opened this bug for discussions > > > > [1]: https://packages.debian.org/sid/rdma-core > > > > rdma-core configures modprobe for ib/rdma modules, installs some > helper daemons and installs some udev rules. It certainly makes > administration of the server an easier task, but it isn't *required* > when configuring a Debian server for rdma and user-space ibverbs. I didn't think it is required for mlx5 as the ethernet driver autoloads the rdma modules itself.. At least, I wouldn't make rdma-core a hard dependency of dpdk.. At worst that should be in libibverbs, as you say. Jason
Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1
On Thu, Mar 21, 2019 at 3:51 AM Christian Ehrhardt wrote: > > > ibverbs is provided by rdma-core project. > > Is it a separate package in your case? > > No it is part of the same src package. > > > Yes, you need libmlx4, libmlx5 and libibverbs. > > It is described in the doc: > > http://doc.dpdk.org/guides/nics/mlx5.html#prerequisites > > Well that is the problem, we fulfill that prereq already. > libmlx4, libmlx5 and libibverbs are installed by dependencies already > and thought all the time that would be good enough. > > But we got told that we also need the package rdma-core [1] itself to > be installed to be useful. > That was a discussion with the subject "Please help us to verify DPDK > 17.11.5 for Ubuntu 18.04/18.10" with Mellanox test people. > Fortunately I keep you on CC on those - so maybe you can find it > somewhere in your inbox and contact the other Mellanox people on that > thread. > After that you might help to translate their request into our terms please? > > The question is: > a) what exactly is (binary package) rdma-core [1] needed for in that scenario > b) if (a) would be generally true we wondered why it had to be a dep > of the DPDK packages and not of e.g. libibverbs1 itself > That is why we opened this bug for discussions > > [1]: https://packages.debian.org/sid/rdma-core > rdma-core configures modprobe for ib/rdma modules, installs some helper daemons and installs some udev rules. It certainly makes administration of the server an easier task, but it isn't *required* when configuring a Debian server for rdma and user-space ibverbs. Perhaps libibverbs should recommend rdma-core? -- Brian T. Smith System Fabric Works Senior Technical Staff bsm...@systemfabricworks.com GPG Key: 0xB3C2C7B73BA3CD7F
Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1
> ibverbs is provided by rdma-core project. > Is it a separate package in your case? No it is part of the same src package. > Yes, you need libmlx4, libmlx5 and libibverbs. > It is described in the doc: > http://doc.dpdk.org/guides/nics/mlx5.html#prerequisites Well that is the problem, we fulfill that prereq already. libmlx4, libmlx5 and libibverbs are installed by dependencies already and thought all the time that would be good enough. But we got told that we also need the package rdma-core [1] itself to be installed to be useful. That was a discussion with the subject "Please help us to verify DPDK 17.11.5 for Ubuntu 18.04/18.10" with Mellanox test people. Fortunately I keep you on CC on those - so maybe you can find it somewhere in your inbox and contact the other Mellanox people on that thread. After that you might help to translate their request into our terms please? The question is: a) what exactly is (binary package) rdma-core [1] needed for in that scenario b) if (a) would be generally true we wondered why it had to be a dep of the DPDK packages and not of e.g. libibverbs1 itself That is why we opened this bug for discussions [1]: https://packages.debian.org/sid/rdma-core
Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1
20/03/2019 12:51, Christian Ehrhardt: > @rdma-core team > - does ibverbs1/ibverbs-providers generally need rdma-core to be useful? ibverbs is provided by rdma-core project. Is it a separate package in your case? > - if so why isn't it at least a recommends from those packages > @Thomas > - do you happen to know what exactly of rdma-core that librte-pmd-mlx[45] > need? Yes, you need libmlx4, libmlx5 and libibverbs. It is described in the doc: http://doc.dpdk.org/guides/nics/mlx5.html#prerequisites
Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1
Package: rdma-core Version: 22.1-1 X-Debbugs-CC: tho...@monjalon.net, bl...@debian.org FYI on CC I have put some more people: CC: Luca who maintains DPDK with me CC: Thomas for DPDK<->Mellanox knowledge that could help Hi, this might be more a request to understand the correct dependencies than a bug (unless you say it is one). But before jumping to conclusions let me outline the issue. We were asked [1] on package DPDK to also install rdma-core as the packages for MLX[4/5] would not work otherwise. We checked our packages and e.g. librte-pmd-mlx5-18.11 depends (due to shlibs magic) on ibverbs-providers and libibverbs1. And we thought that would be fine. Now after being asked/told that without rdma-core this would be useless we were wondering. rdma-core is essentially a bunch of module config/autoloaders and rdma-ndd/iwpmd services. Right now we can only assume this is what librte-pmd-mlx5-18.11 needs, as we are lacking some details. Therefore the following questions: @rdma-core team - does ibverbs1/ibverbs-providers generally need rdma-core to be useful? - if so why isn't it at least a recommends from those packages @Thomas - do you happen to know what exactly of rdma-core that librte-pmd-mlx[45] need? looking forward to your reply, as I'm happy to learn more about the dependencies among those packages. [1]: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1820957 (I created that bug from a mail thread) -- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd
