NRS only affects Lustre traffic, so it will not factor into lnet_selftest (LST) results.
I gave some talks on troubleshooting multi-rail that you may want to review. Overview: https://youtu.be/j3m-mznUdac?feature=shared Demo: https://youtu.be/TLN56cw9Zgs?feature=shared You should probably start by verifying that the client and server see each other as multi-rail peers, and by checking the send and receive counts for each interface on your client and server to ensure that traffic is being spread across them. Chris Horn From: lustre-discuss <[email protected]> on behalf of Gwen Dawes via lustre-discuss <[email protected]> Date: Wednesday, January 17, 2024 at 5:48 AM To: [email protected] <[email protected]> Subject: Re: [lustre-discuss] LNet Multi-Rail config - with BODY! Hi Andreas, Thanks for the pointer. I have a second server set up running 2.15.3 as well specifically to check this, and can set it up with lnet_selftest, same as the client. After taking a bit to convince the fabric manager to accept the moved IPs, I get the exact same results between the two. Good to know that it is possible, though - I wonder what needs to be modified to achieve that. It's completely stock - the UDSP is just blank, and the default NRS config is in play. I don't suppose there's any chance the NRS config is what I'm missing? Gwen On Wed, 2024-01-17 at 03:14 +0000, Andreas Dilger wrote: > Hello Gwen, > I'm not a networking expert, but it seems entirely possible that the > MR discovery in 2.12.9 > isn't doing as well as what is in 2.15.3 (or 2.15.4 for that matter). > It would make more sense > to have both nodes running the same (newer) version before digging > too deeply into this. > > We have definitely seen performance > 1 IB interface from a single > node in our testing, > though I can't say if that was done with lnet_selftest or with > something else. > > Cheers, Andreas > > > On Jan 16, 2024, at 08:14, Gwen Dawes via lustre-discuss > > <[email protected]> wrote: > > > > Hi folks, > > > > Let's try that again. > > > > I'm in the luxury position of having four IB cards I'm trying to > > squeeze the most performance out of for Lustre I can. > > > > I have a small test setup - two machines - a client (2.12.9) and a > > server (2.15.3) with four IB cards each. I'm able to set them up as > > Multi-Rail and each one can discover the other as such. However, I > > can't seem to get lnet_selftest to give me more speed than a single > > interface, as reported by ib_send_bw. > > > > Am I missing some config here? Is LNet just not capable of doing > > more > > than one connection per NID? > > > > Gwen > > _______________________________________________ > > lustre-discuss mailing list > > [email protected] > > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org> > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Whamcloud > > > > > > > _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org>
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
