Local LNet configuration can be done either via modprobe config or via lnetctl/yaml. We are slowly moving away from modprobe config (kernel module parameters) in favor of lnetctl/yaml because the latter provides more flexibility.
For IB and TCP networks, every interface needs an IP address assigned. It is okay to have multiple interfaces on the same subnet as long as you have appropriate ip route/rules and ARP settings in place. Otherwise the network stack may not actually send traffic to/from the correct interfaces, or there may be connection failures, etc. There was some work to do this automatically for TCP networks in https://jira.whamcloud.com/browse/LU-14662 . There is some discussion of the issue on the wiki at https://wiki.lustre.org/LNet_Router_Config_Guide#ARP_flux_issue_for_MR_node but I’m not sure how up-to-date that guidance is. LNet/ko2iblnd only uses IPoIB for connection setup via RDMA CM. After a connection is established all traffic between IB peers is via RDMA protocol. The multi-rail feature requires more than just a local LNet configuration. It also requires configuration of the peer table. In Lustre 2.10, this peer table was statically defined. In Lustre 2.11 (and later), the LNet Dynamic Peer Discovery feature allows LNet to create the peer table dynamically.= Chris Horn From: lustre-discuss <[email protected]> on behalf of Åke Sandgren <[email protected]> Date: Monday, January 17, 2022 at 1:10 AM To: Lustre discussion <[email protected]> Subject: Re: [lustre-discuss] IPoIB best practises On 1/17/22 2:36 AM, Angelos Ching via lustre-discuss wrote: > Hi Eli, > > Yes & no; part of my info is a bit rusty because I carried them from > version around 2.10. MR is now turned on by default. > > But you'll need to have an IP setup on each IPoIB interface, and for all > ib0 & all ib1 interface, they should be in different subnet. Eg: all ib0 > on 192.168.100.0/24 and all ib1 on 192.168.101.0/24 The multirail setup we have is that both ib0 and ib1 are on the same subnet, that's how DDN configured it for us. ip a s ib0 | grep inet inet 172.27.1.30/24 brd 172.27.1.255 scope global ib0 ip a s ib1 | grep inet inet 172.27.1.50/24 brd 172.27.1.255 scope global ib1 and the modprobe config is options lnet networks="o2ib1(ib0,ib1)" -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: [email protected] Phone: +46 90 7866134 Fax: +46 90-580 14 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se<http://www.hpc2n.umu.se> _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org>
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
