subject:"Re\: \[lustre\-discuss\] LNet Multi\-Rail config"

Re: [lustre-discuss] LNet Multi-Rail config

2024-02-22 Thread Horn, Chris via lustre-discuss

Can you share the client’s cpt configuration?

$ lctl get_param cpu_partition_table cpu_partition_distance

Chris Horn

From: lustre-discuss  on behalf of 
Gwen Dawes via lustre-discuss 
Date: Wednesday, February 14, 2024 at 11:19 AM
To: lustre-discuss@lists.lustre.org 
Subject: Re: [lustre-discuss] LNet Multi-Rail config
Hi Chris,

Thanks for the pointers - apologies about kicking up an old thread, but
I'm running out of ideas for how to solve this one.

I switched everything out to 2.15.4 and carefully documented the build
process, then turned one of my machines into a VM host with PCI
passthrough to eliminate any NUMA issues and additional complexity.

So now I have a much simpler layout - one client, multi-rail, four
interfaces, with four servers (1 interface each), one of which put
aside for coordinating lnet_selftest runs. Everything sees everything
else as a Multi-Rail peer, with 1 interface for my servers and 4 for my
client.

My client has two CPTs set up - with the cards on each CPU set up into
differing 'dev cpt' numbers - 0 and 1 - as per their bus.

Running an lnet selftest of 'read', though (concurrency 32, simple
check) - even with distribute 1:3 set, I always see exactly two
interfaces in use. I can block them with net_drop_add which eventually
forces the traffic off to the others, but it only ever seems to use two
interfaces.

Is this a bug with lnet_selftest? Some kind of non-network bottleneck?

Am I misunderstanding CPTs?

Gwen


On Wed, 2024-01-17 at 17:53 +, Horn, Chris wrote:
> NRS only affects Lustre traffic, so it will not factor into
> lnet_selftest (LST) results.
>
> I gave some talks on troubleshooting multi-rail that you may want to
> review.
> Overview:
> https://youtu.be/j3m-mznUdac?feature=shared<https://youtu.be/j3m-mznUdac?feature=shared>
> Demo:
> https://youtu.be/TLN56cw9Zgs?feature=shared<https://youtu.be/TLN56cw9Zgs?feature=shared>
>
> You should probably start by verifying that the client and server see
> each other as multi-rail peers, and by checking the send and receive
> counts for each interface on your client and server to ensure that
> traffic is being spread across them.
>
> Chris Horn
>
> From:lustre-discuss  on
> behalf of Gwen Dawes via lustre-discuss
> 
> Date: Wednesday, January 17, 2024 at 5:48 AM
> To: lustre-discuss@lists.lustre.org 
> Subject: Re: [lustre-discuss] LNet Multi-Rail config - with BODY!
> Hi Andreas,
>
> Thanks for the pointer. I have a second server set up running 2.15.3
> as
> well specifically to check this, and can set it up with
> lnet_selftest,
> same as the client. After taking a bit to convince the fabric manager
> to accept the moved IPs, I get the exact same results between the
> two.
>
> Good to know that it is possible, though - I wonder what needs to be
> modified to achieve that. It's completely stock - the UDSP is just
> blank, and the default NRS config is in play.
>
> I don't suppose there's any chance the NRS config is what I'm
> missing?
>
> Gwen
>
> On Wed, 2024-01-17 at 03:14 +, Andreas Dilger wrote:
> > Hello Gwen,
> > I'm not a networking expert, but it seems entirely possible that
> > the
> > MR discovery in 2.12.9
> > isn't doing as well as what is in 2.15.3 (or 2.15.4 for that
> > matter).
> >  It would make more sense
> > to have both nodes running the same (newer) version before digging
> > too deeply into this.
> >
> > We have definitely seen performance > 1 IB interface from a single
> > node in our testing,
> > though I can't say if that was done with lnet_selftest or with
> > something else.
> >
> > Cheers, Andreas
> >
> > > On Jan 16, 2024, at 08:14, Gwen Dawes via lustre-discuss
> > >  wrote:
> > >
> > > Hi folks,
> > >
> > > Let's try that again.
> > >
> > > I'm in the luxury position of having four IB cards I'm trying to
> > > squeeze the most performance out of for Lustre I can.
> > >
> > > I have a small test setup - two machines - a client (2.12.9) and
> > > a
> > > server (2.15.3) with four IB cards each. I'm able to set them up
> > > as
> > > Multi-Rail and each one can discover the other as such. However,
> > > I
> > > can't seem to get lnet_selftest to give me more speed than a
> > > single
> > > interface, as reported by ib_send_bw.
> > >
> > > Am I missing some config here? Is LNet just not capable of doing
> > > more
> > > than one connection per NID?
> > >
> > > Gwen
> > > ___
> &

Re: [lustre-discuss] LNet Multi-Rail config

2024-02-14 Thread Gwen Dawes via lustre-discuss

Hi Chris,

Thanks for the pointers - apologies about kicking up an old thread, but
I'm running out of ideas for how to solve this one.

I switched everything out to 2.15.4 and carefully documented the build
process, then turned one of my machines into a VM host with PCI
passthrough to eliminate any NUMA issues and additional complexity.

So now I have a much simpler layout - one client, multi-rail, four
interfaces, with four servers (1 interface each), one of which put
aside for coordinating lnet_selftest runs. Everything sees everything
else as a Multi-Rail peer, with 1 interface for my servers and 4 for my
client.

My client has two CPTs set up - with the cards on each CPU set up into
differing 'dev cpt' numbers - 0 and 1 - as per their bus.

Running an lnet selftest of 'read', though (concurrency 32, simple
check) - even with distribute 1:3 set, I always see exactly two
interfaces in use. I can block them with net_drop_add which eventually
forces the traffic off to the others, but it only ever seems to use two
interfaces.

Is this a bug with lnet_selftest? Some kind of non-network bottleneck?

Am I misunderstanding CPTs?

Gwen


On Wed, 2024-01-17 at 17:53 +, Horn, Chris wrote:
> NRS only affects Lustre traffic, so it will not factor into
> lnet_selftest (LST) results.
>  
> I gave some talks on troubleshooting multi-rail that you may want to
> review.
> Overview:
> https://youtu.be/j3m-mznUdac?feature=shared
> Demo:
> https://youtu.be/TLN56cw9Zgs?feature=shared
>  
> You should probably start by verifying that the client and server see
> each other as multi-rail peers, and by checking the send and receive
> counts for each interface on your client and server to ensure that
> traffic is being spread across them.
>  
> Chris Horn
>  
> From:lustre-discuss  on
> behalf of Gwen Dawes via lustre-discuss
> 
> Date: Wednesday, January 17, 2024 at 5:48 AM
> To: lustre-discuss@lists.lustre.org 
> Subject: Re: [lustre-discuss] LNet Multi-Rail config - with BODY!
> Hi Andreas,
> 
> Thanks for the pointer. I have a second server set up running 2.15.3
> as
> well specifically to check this, and can set it up with
> lnet_selftest,
> same as the client. After taking a bit to convince the fabric manager
> to accept the moved IPs, I get the exact same results between the
> two.
> 
> Good to know that it is possible, though - I wonder what needs to be
> modified to achieve that. It's completely stock - the UDSP is just
> blank, and the default NRS config is in play.
> 
> I don't suppose there's any chance the NRS config is what I'm
> missing?
> 
> Gwen
> 
> On Wed, 2024-01-17 at 03:14 +, Andreas Dilger wrote:
> > Hello Gwen,
> > I'm not a networking expert, but it seems entirely possible that
> > the
> > MR discovery in 2.12.9
> > isn't doing as well as what is in 2.15.3 (or 2.15.4 for that
> > matter).
> >  It would make more sense
> > to have both nodes running the same (newer) version before digging
> > too deeply into this.
> > 
> > We have definitely seen performance > 1 IB interface from a single
> > node in our testing,
> > though I can't say if that was done with lnet_selftest or with
> > something else.
> > 
> > Cheers, Andreas
> > 
> > > On Jan 16, 2024, at 08:14, Gwen Dawes via lustre-discuss
> > >  wrote:
> > > 
> > > Hi folks,
> > > 
> > > Let's try that again.
> > > 
> > > I'm in the luxury position of having four IB cards I'm trying to
> > > squeeze the most performance out of for Lustre I can.
> > > 
> > > I have a small test setup - two machines - a client (2.12.9) and
> > > a
> > > server (2.15.3) with four IB cards each. I'm able to set them up
> > > as
> > > Multi-Rail and each one can discover the other as such. However,
> > > I
> > > can't seem to get lnet_selftest to give me more speed than a
> > > single
> > > interface, as reported by ib_send_bw.
> > > 
> > > Am I missing some config here? Is LNet just not capable of doing
> > > more
> > > than one connection per NID?
> > > 
> > > Gwen
> > > ___
> > > lustre-discuss mailing list
> > > lustre-discuss@lists.lustre.org
> > > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> > 
> > Cheers, Andreas
> > --
> > Andreas Dilger
> > Lustre Principal Architect
> > Whamcloud
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> 
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] LNet Multi-Rail config - with BODY!

2024-01-17 Thread Horn, Chris via lustre-discuss

NRS only affects Lustre traffic, so it will not factor into lnet_selftest (LST) 
results.

I gave some talks on troubleshooting multi-rail that you may want to review.
Overview:
https://youtu.be/j3m-mznUdac?feature=shared
Demo:
https://youtu.be/TLN56cw9Zgs?feature=shared

You should probably start by verifying that the client and server see each 
other as multi-rail peers, and by checking the send and receive counts for each 
interface on your client and server to ensure that traffic is being spread 
across them.

Chris Horn

From: lustre-discuss  on behalf of 
Gwen Dawes via lustre-discuss 
Date: Wednesday, January 17, 2024 at 5:48 AM
To: lustre-discuss@lists.lustre.org 
Subject: Re: [lustre-discuss] LNet Multi-Rail config - with BODY!
Hi Andreas,

Thanks for the pointer. I have a second server set up running 2.15.3 as
well specifically to check this, and can set it up with lnet_selftest,
same as the client. After taking a bit to convince the fabric manager
to accept the moved IPs, I get the exact same results between the two.

Good to know that it is possible, though - I wonder what needs to be
modified to achieve that. It's completely stock - the UDSP is just
blank, and the default NRS config is in play.

I don't suppose there's any chance the NRS config is what I'm missing?

Gwen

On Wed, 2024-01-17 at 03:14 +, Andreas Dilger wrote:
> Hello Gwen,
> I'm not a networking expert, but it seems entirely possible that the
> MR discovery in 2.12.9
> isn't doing as well as what is in 2.15.3 (or 2.15.4 for that matter).
>  It would make more sense
> to have both nodes running the same (newer) version before digging
> too deeply into this.
>
> We have definitely seen performance > 1 IB interface from a single
> node in our testing,
> though I can't say if that was done with lnet_selftest or with
> something else.
>
> Cheers, Andreas
>
> > On Jan 16, 2024, at 08:14, Gwen Dawes via lustre-discuss
> >  wrote:
> >
> > Hi folks,
> >
> > Let's try that again.
> >
> > I'm in the luxury position of having four IB cards I'm trying to
> > squeeze the most performance out of for Lustre I can.
> >
> > I have a small test setup - two machines - a client (2.12.9) and a
> > server (2.15.3) with four IB cards each. I'm able to set them up as
> > Multi-Rail and each one can discover the other as such. However, I
> > can't seem to get lnet_selftest to give me more speed than a single
> > interface, as reported by ib_send_bw.
> >
> > Am I missing some config here? Is LNet just not capable of doing
> > more
> > than one connection per NID?
> >
> > Gwen
> > ___
> > lustre-discuss mailing list
> > lustre-discuss@lists.lustre.org
> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org>
>
> Cheers, Andreas
> --
> Andreas Dilger
> Lustre Principal Architect
> Whamcloud
>
>
>
>
>
>
>

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] LNet Multi-Rail config - with BODY!

2024-01-17 Thread Gwen Dawes via lustre-discuss

Hi Andreas,

Thanks for the pointer. I have a second server set up running 2.15.3 as
well specifically to check this, and can set it up with lnet_selftest,
same as the client. After taking a bit to convince the fabric manager
to accept the moved IPs, I get the exact same results between the two.

Good to know that it is possible, though - I wonder what needs to be
modified to achieve that. It's completely stock - the UDSP is just
blank, and the default NRS config is in play.

I don't suppose there's any chance the NRS config is what I'm missing?

Gwen

On Wed, 2024-01-17 at 03:14 +, Andreas Dilger wrote:
> Hello Gwen,
> I'm not a networking expert, but it seems entirely possible that the
> MR discovery in 2.12.9
> isn't doing as well as what is in 2.15.3 (or 2.15.4 for that matter).
>  It would make more sense
> to have both nodes running the same (newer) version before digging
> too deeply into this.
> 
> We have definitely seen performance > 1 IB interface from a single
> node in our testing,
> though I can't say if that was done with lnet_selftest or with
> something else.
> 
> Cheers, Andreas
> 
> > On Jan 16, 2024, at 08:14, Gwen Dawes via lustre-discuss
> >  wrote:
> > 
> > Hi folks,
> > 
> > Let's try that again.
> > 
> > I'm in the luxury position of having four IB cards I'm trying to
> > squeeze the most performance out of for Lustre I can.
> > 
> > I have a small test setup - two machines - a client (2.12.9) and a
> > server (2.15.3) with four IB cards each. I'm able to set them up as
> > Multi-Rail and each one can discover the other as such. However, I
> > can't seem to get lnet_selftest to give me more speed than a single
> > interface, as reported by ib_send_bw.
> > 
> > Am I missing some config here? Is LNet just not capable of doing
> > more
> > than one connection per NID?
> > 
> > Gwen
> > ___
> > lustre-discuss mailing list
> > lustre-discuss@lists.lustre.org
> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Lustre Principal Architect
> Whamcloud
> 
> 
> 
> 
> 
> 
> 

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] LNet Multi-Rail config - with BODY!

2024-01-16 Thread Andreas Dilger via lustre-discuss

Hello Gwen,
I'm not a networking expert, but it seems entirely possible that the MR 
discovery in 2.12.9
isn't doing as well as what is in 2.15.3 (or 2.15.4 for that matter).  It would 
make more sense
to have both nodes running the same (newer) version before digging too deeply 
into this.

We have definitely seen performance > 1 IB interface from a single node in our 
testing,
though I can't say if that was done with lnet_selftest or with something else.

Cheers, Andreas

On Jan 16, 2024, at 08:14, Gwen Dawes via lustre-discuss 
mailto:lustre-discuss@lists.lustre.org>> wrote:

Hi folks,

Let's try that again.

I'm in the luxury position of having four IB cards I'm trying to
squeeze the most performance out of for Lustre I can.

I have a small test setup - two machines - a client (2.12.9) and a
server (2.15.3) with four IB cards each. I'm able to set them up as
Multi-Rail and each one can discover the other as such. However, I
can't seem to get lnet_selftest to give me more speed than a single
interface, as reported by ib_send_bw.

Am I missing some config here? Is LNet just not capable of doing more
than one connection per NID?

Gwen
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud







___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] LNet Multi-Rail config

Re: [lustre-discuss] LNet Multi-Rail config

Re: [lustre-discuss] LNet Multi-Rail config - with BODY!

Re: [lustre-discuss] LNet Multi-Rail config - with BODY!

Re: [lustre-discuss] LNet Multi-Rail config - with BODY!

5 matches

Site Navigation

Mail list logo

Footer information