Re: [lustre-discuss] 2.15.4 o2iblnd on RoCEv2?

[Andreas Dilger via lustre-discuss]

Granted that I'm not an LNet expert, but "errno: -1 descr: cannot parse net 
'<255:65535>' " doesn't immediately lead me to the same conclusion as if 
"unknown internface 'ib0' " were printed for the error message.  Also "errno: 
-1" is "-EPERM = Operation not permitted", and doesn't give the same 
information as "-ENXIO = No such device or address" or even "-EINVAL = Invalid 
argument" would.

That said, I can't even offer a patch for this myself, since that exact error 
message is used in a few different places, though I suspect it is coming from 
lustre_lnet_config_ni().

Looking further into this, now that I've found where (I think) the error 
message is generated, it seems that "errno: -1" is not "-EPERM" but rather 
"LUSTRE_CFG_RC_BAD_PARAM", which is IMHO a travesty to use different error 
numbers (and then print them after "errno:") instead of existing POSIX error 
codes that could fill the same role (with some creative mapping):

#define LUSTRE_CFG_RC_NO_ERR 0  => fine
#define LUSTRE_CFG_RC_BAD_PARAM -1  => -EINVAL
#define LUSTRE_CFG_RC_MISSING_PARAM -2  => -EFAULT
#define LUSTRE_CFG_RC_OUT_OF_RANGE_PARAM-3  => -ERANGE
#define LUSTRE_CFG_RC_OUT_OF_MEM-4  => -ENOMEM
#define LUSTRE_CFG_RC_GENERIC_ERR   -5  => -ENODATA
#define LUSTRE_CFG_RC_NO_MATCH  -6  => -ENOMSG
#define LUSTRE_CFG_RC_MATCH -7  => -EXFULL
#define LUSTRE_CFG_RC_SKIP  -8  => -EBADSLT
#define LUSTRE_CFG_RC_LAST_ELEM -9  => -ECHRNG
#define LUSTRE_CFG_RC_MARSHAL_FAIL  -10 => -ENOSTR

I don't think "overloading" the POSIX error codes to mean something similar is 
worse than using random numbers to report errors.  Also, in some cases (even in 
lustre_lnet_config_ni()) it is using "rc = -errno" so the LUSTRE_CFG_RC_* 
errors are *already* conflicting with POSIX error numbers, and it impossible to 
distinguish between them...

The main question is whether changing these numbers will break a user->kernel 
interface, or if these definitions are only in userspace?It looks like 
lnetctl.c is only ever checking "!= LUSTRE_CFG_RC_NO_ERR", so maybe it is fine? 
 None of the values currently overlap, so it would be possible to start 
accepting either of the values for the return in the user tools, and then at 
some point in the future start actually returning them...  Something for the 
LNet folks to figure out.

Cheers, Andreas

On Jan 10, 2024, at 13:29, Jeff Johnson 
mailto:jeff.john...@aeoncomputing.com>> wrote:

A LU ticket and patch for lnetctl or for me being an under-caffeinated
idiot? ;-)

On Wed, Jan 10, 2024 at 12:06 PM Andreas Dilger 
mailto:adil...@whamcloud.com>> wrote:

It would seem that the error message could be improved in this case?  Could you 
file an LU ticket for that with the reproducer below, and ideally along with a 
patch?

Cheers, Andreas
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud







___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] 2.15.4 o2iblnd on RoCEv2?

[Jeff Johnson]

A LU ticket and patch for lnetctl or for me being an under-caffeinated
idiot? ;-)

On Wed, Jan 10, 2024 at 12:06 PM Andreas Dilger  wrote:
>
> It would seem that the error message could be improved in this case?  Could 
> you file an LU ticket for that with the reproducer below, and ideally along 
> with a patch?
>
> Cheers, Andreas
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] 2.15.4 o2iblnd on RoCEv2?

[Andreas Dilger via lustre-discuss]

It would seem that the error message could be improved in this case?  Could you 
file an LU ticket for that with the reproducer below, and ideally along with a 
patch?

Cheers, Andreas

> On Jan 10, 2024, at 11:37, Jeff Johnson  
> wrote:
> 
> Man am I an idiot. Been up all night too many nights in a row and not
> enough coffee. It helps if you use the correct --net designation. I
> was typing ib0 instead of o2ib0. Declaring as o2ib0 works fine.
> 
> (cleanup from previous)
> lctl net down && lustre_rmmod
> 
> (new attempt)
> modprobe lnet -v
> lnetctl lnet configure
> lnetctl net add --if enp1s0np0 --net o2ib0
> lnetctl net show
> net:
>- net type: lo
>  local NI(s):
>- nid: 0@lo
>  status: up
>- net type: o2ib
>  local NI(s):
>- nid: 10.0.50.27@o2ib
>  status: up
>  interfaces:
>  0: enp1s0np0
> 
> Lots more to test and verify but the original mailing list submission
> was total pilot error on my part. Apologies to all who spent cycles
> pondering this nothingburger.
> 
> 
> 
> 
>> On Tue, Jan 9, 2024 at 7:45 PM Jeff Johnson
>>  wrote:
>> 
>> Howdy intrepid Lustrefarians,
>> 
>> While starting down the debug rabbit hole I thought I'd raise my hand
>> and see if anyone has a few magic beans to spare.
>> 
>> I cannot get lnet (via lnetctl) to init a o2iblnd interface on a
>> RoCEv2 interface.
>> 
>> Running `lnetctl net add --net ib0 --if enp1s0np0` results in
>> net:
>>  errno: -1
>>  descr: cannot parse net '<255:65535>'
>> 
>> Nothing in dmesg to indicate why. Search engines aren't coughing up
>> much here either.
>> 
>> Env: Rocky 8.9 x86_64, MOFED 5.8-4.1.5.0, Lustre 2.15.4
>> 
>> I'm able to run mpi over the RoCEv2 interface. Utils like ibstatus and
>> ibdev2netdev report it correctly. ibv_rc_pingpong works fine between
>> nodes.
>> 
>> Configuring as socklnd works fine. `lnetctl net add --net tcp0 --if
>> enp1s0np0 && lnetctl net show`
>> [root@r2u11n3 ~]# lnetctl net show
>> net:
>>- net type: lo
>>  local NI(s):
>>- nid: 0@lo
>>  status: up
>>- net type: tcp
>>  local NI(s):
>>- nid: 10.0.50.27@tcp
>>  status: up
>>  interfaces:
>>  0: enp1s0np0
>> 
>> I verified the RoCEv2 interface using nVidia's `cma_roce_mode` as well
>> as sysfs references
>> 
>> [root@r2u11n3 ~]# cma_roce_mode -d mlx5_0 -p 1
>> RoCE v2
>> 
>> Ideas? Suggestions? Incense?
>> 
>> Thanks,
>> 
>> --Jeff
> 
> 
> 
> --
> --
> Jeff Johnson
> Co-Founder
> Aeon Computing
> 
> jeff.john...@aeoncomputing.com
> www.aeoncomputing.com
> t: 858-412-3810 x1001   f: 858-412-3845
> m: 619-204-9061
> 
> 4170 Morena Boulevard, Suite C - San Diego, CA 92117
> 
> High-Performance Computing / Lustre Filesystems / Scale-out Storage
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] 2.15.4 o2iblnd on RoCEv2?

[Jeff Johnson]

Man am I an idiot. Been up all night too many nights in a row and not
enough coffee. It helps if you use the correct --net designation. I
was typing ib0 instead of o2ib0. Declaring as o2ib0 works fine.

(cleanup from previous)
lctl net down && lustre_rmmod

(new attempt)
modprobe lnet -v
lnetctl lnet configure
lnetctl net add --if enp1s0np0 --net o2ib0
lnetctl net show
net:
- net type: lo
  local NI(s):
- nid: 0@lo
  status: up
- net type: o2ib
  local NI(s):
- nid: 10.0.50.27@o2ib
  status: up
  interfaces:
  0: enp1s0np0

Lots more to test and verify but the original mailing list submission
was total pilot error on my part. Apologies to all who spent cycles
pondering this nothingburger.




On Tue, Jan 9, 2024 at 7:45 PM Jeff Johnson
 wrote:
>
> Howdy intrepid Lustrefarians,
>
> While starting down the debug rabbit hole I thought I'd raise my hand
> and see if anyone has a few magic beans to spare.
>
> I cannot get lnet (via lnetctl) to init a o2iblnd interface on a
> RoCEv2 interface.
>
> Running `lnetctl net add --net ib0 --if enp1s0np0` results in
>  net:
>   errno: -1
>   descr: cannot parse net '<255:65535>'
>
> Nothing in dmesg to indicate why. Search engines aren't coughing up
> much here either.
>
> Env: Rocky 8.9 x86_64, MOFED 5.8-4.1.5.0, Lustre 2.15.4
>
> I'm able to run mpi over the RoCEv2 interface. Utils like ibstatus and
> ibdev2netdev report it correctly. ibv_rc_pingpong works fine between
> nodes.
>
> Configuring as socklnd works fine. `lnetctl net add --net tcp0 --if
> enp1s0np0 && lnetctl net show`
> [root@r2u11n3 ~]# lnetctl net show
> net:
> - net type: lo
>   local NI(s):
> - nid: 0@lo
>   status: up
> - net type: tcp
>   local NI(s):
> - nid: 10.0.50.27@tcp
>   status: up
>   interfaces:
>   0: enp1s0np0
>
> I verified the RoCEv2 interface using nVidia's `cma_roce_mode` as well
> as sysfs references
>
> [root@r2u11n3 ~]# cma_roce_mode -d mlx5_0 -p 1
> RoCE v2
>
> Ideas? Suggestions? Incense?
>
> Thanks,
>
> --Jeff



-- 
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite C - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] 2.15.4 o2iblnd on RoCEv2?

[Jeff Johnson]

Howdy intrepid Lustrefarians,

While starting down the debug rabbit hole I thought I'd raise my hand
and see if anyone has a few magic beans to spare.

I cannot get lnet (via lnetctl) to init a o2iblnd interface on a
RoCEv2 interface.

Running `lnetctl net add --net ib0 --if enp1s0np0` results in
 net:
  errno: -1
  descr: cannot parse net '<255:65535>'

Nothing in dmesg to indicate why. Search engines aren't coughing up
much here either.

Env: Rocky 8.9 x86_64, MOFED 5.8-4.1.5.0, Lustre 2.15.4

I'm able to run mpi over the RoCEv2 interface. Utils like ibstatus and
ibdev2netdev report it correctly. ibv_rc_pingpong works fine between
nodes.

Configuring as socklnd works fine. `lnetctl net add --net tcp0 --if
enp1s0np0 && lnetctl net show`
[root@r2u11n3 ~]# lnetctl net show
net:
- net type: lo
  local NI(s):
- nid: 0@lo
  status: up
- net type: tcp
  local NI(s):
- nid: 10.0.50.27@tcp
  status: up
  interfaces:
  0: enp1s0np0

I verified the RoCEv2 interface using nVidia's `cma_roce_mode` as well
as sysfs references

[root@r2u11n3 ~]# cma_roce_mode -d mlx5_0 -p 1
RoCE v2

Ideas? Suggestions? Incense?

Thanks,

--Jeff
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org