Re: [lustre-discuss] [EXTERNAL] Mounting lustre on block device

[Andreas Dilger via lustre-discuss]

On Mar 16, 2023, at 17:02, Jeff Johnson 
mailto:jeff.john...@aeoncomputing.com>> wrote:

If you *really* want a block device on a client that resides in Lustre you 
*could* create a file in Lustre and then make that file a loopback device with 
losetup. Of course, your mileage will vary *a lot* based on use case, access, 
underlying LFS configuration.

dd if=/dev/zero of=/my_lustre_mountpoint/some_subdir/big_raw_file bs=1048576 
count=10
losetup -f /my_lustre_mountpoint/some_subdir/big_raw_file
*assuming loop0 is created*
some_fun_command /dev/loop0

Note with ldiskfs backends you can use "fallocate -l 10M 
/my_lustre_mountpoint/some_subdir/big_raw_file" to reserve the space.

Alternately, if you have flash-based OSTs you could truncate a sparse file to 
the full size ("truncate -S 10M ...") and format that, which will not 
consume as much space but will generate more random allocation on the OSTs.

Disclaimer: Just because you *can* do this, doesn't necessarily mean it is a 
good idea

We saw a good performance boost with ext4 images on Lustre holding many small 
files (CCI).  Also, I recall some customers in the past using ext2 or ext4 
images effectively to aggregate many small files for read-only use on compute 
nodes.

Cheers, Andreas


On Thu, Mar 16, 2023 at 3:29 PM Mohr, Rick via lustre-discuss 
mailto:lustre-discuss@lists.lustre.org>> wrote:
Are you asking if you can mount Lustre on a client so that it shows up as a 
block device?  If so, the answer to that is you can't.  Lustre does not appear 
as a block device to the clients.

-Rick



On 3/16/23, 3:44 PM, "lustre-discuss on behalf of Shambhu Raje via 
lustre-discuss" 
mailto:lustre-discuss-boun...@lists.lustre.org>
 
>
 on behalf of 
lustre-discuss@lists.lustre.org 
>>
 wrote:


When we mount a lustre file system on client, the lustre file system does not 
use block device on client side. Instead it uses virtual file system namespace. 
Mounting point will not be shown when we do 'lsblk'. As it only show on 'df-hT'.


How can we mount lustre file system on block such that when we write something 
with lusterfs then it can be shown in block device??
Can share command??











___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


--
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite C - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud







___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lnet config serving multiple routers and clients

[Kumar, Amit via lustre-discuss]

Yes I do but in the same state as server it is down. Also I have set routing to 
1 on lnet;

# lnetctl route show -v
route:
- net: o2ib
  gateway: 10.215.25.76@o2ib2
  hop: -1
  priority: 0
  health_sensitivity: 1
  state: down
  type: single-hop

From: Horn, Chris 
Sent: Wednesday, April 5, 2023 12:33 PM
To: Kumar, Amit ; lustre-discuss@lists.lustre.org
Subject: Re: Lnet config serving multiple routers and clients


[EXTERNAL SENDER]
Do you have the route to o2ib via 10.215.25.76@o2ib2 
defined on the client?

Chris Horn

From: lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 on behalf of Kumar, Amit via lustre-discuss 
mailto:lustre-discuss@lists.lustre.org>>
Date: Wednesday, April 5, 2023 at 12:28 PM
To: lustre-discuss@lists.lustre.org 
mailto:lustre-discuss@lists.lustre.org>>
Subject: [lustre-discuss] Lnet config serving multiple routers and clients

Dear Lustre team,



Below lustre server(showing one of many) is already serving current file system 
via lnet routers over o2ib1 to another cluster;



Now we are adding a new router to serve another new cluster and its clients 
over o2ib2;



Apparently, I can communicate via ping to lnet router's both o2ib and o2ib2 
NIDs; Likewise, from client I can ping both o2ib and o2ib2 NIDs on lnet router. 
But end to end communication between client and server cannot find route to 
each other.



Initially I thought it me be related to LU-11641, given I can access both NIDs 
on the immediate peer I am guessing it is something in my config. I wanted to 
see if a second set of eyes could point out what could I be doing wrong. Any 
idea?



Server is @ lustre-2.12.5-1.el7.x86_64 on CentOS7.8;

Lnet router is @ lustre-client-2.12.5-1.el7.x86_64 on CentOS7.8



Client is @lustre-client-2.14.0 on Ubuntu 22.04***;



Server(10.212.14.9@o2ib)

Lnet router(10.212.1.11@o2ib & 
10.215.25.76@o2ib2)

Client (10.215.25.74@o2ib2)



** Server: **

# lnetctl net show

net:

- net type: lo

  local NI(s):

- nid: 0@lo

  status: up

- net type: o2ib

  local NI(s):

- nid: 10.212.14.9@o2ib

  status: up

  interfaces:

  0: ib0



# lnetctl route show

route:

- net: o2ib1

  gateway: 10.212.15.16@o2ib

- net: o2ib1

  gateway: 10.212.16.20@o2ib

- net: o2ib2

  gateway: 10.212.1.11@o2ib



# lnetctl ping 10.212.1.11@o2ib

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.215.25.76@o2ib2

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.215.25.74@o2ib2

manage:

- ping:

  errno: -1

  descr: failed to ping 10.215.25.74@o2ib2: 
Input/output error



LNET router 

# lnetctl net show

net:

- net type: lo

  local NI(s):

- nid: 0@lo

  status: up

- net type: o2ib

  local NI(s):

- nid: 10.212.1.11@o2ib

  status: up

  interfaces:

  0: ib1

- net type: o2ib2

  local NI(s):

- nid: 10.215.25.76@o2ib2

  status: down

  interfaces:

  0: ib0





# lnetctl discover 10.212.1.11@o2ib

discover:

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl discover 10.215.25.76@o2ib2

discover:

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.212.14.9@o2ib

ping:

- primary nid: 10.212.14.9@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.14.9@o2ib



# lnetctl ping 10.215.25.75@o2ib2

ping:

- primary nid: 10.215.25.75@o2ib2

  Multi-Rail: True

  peer ni:

- n

Re: [lustre-discuss] Lnet config serving multiple routers and clients

[Kumar, Amit via lustre-discuss]

Addendum:
Route shows down no idea how to plumb this.

# lnetctl route show -v
route:
- net: o2ib1
  gateway: 10.212.15.16@o2ib
  hop: -1
  priority: 0
  state: up
- net: o2ib1
  gateway: 10.212.16.20@o2ib
  hop: -1
  priority: 0
  state: up
- net: o2ib2
  gateway: 10.212.1.11@o2ib
  hop: -1
  priority: 0
  state: down

From: lustre-discuss  On Behalf Of 
Kumar, Amit via lustre-discuss
Sent: Wednesday, April 5, 2023 12:27 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] Lnet config serving multiple routers and clients


[EXTERNAL SENDER]

Dear Lustre team,



Below lustre server(showing one of many) is already serving current file system 
via lnet routers over o2ib1 to another cluster;



Now we are adding a new router to serve another new cluster and its clients 
over o2ib2;



Apparently, I can communicate via ping to lnet router's both o2ib and o2ib2 
NIDs; Likewise, from client I can ping both o2ib and o2ib2 NIDs on lnet router. 
But end to end communication between client and server cannot find route to 
each other.



Initially I thought it me be related to LU-11641, given I can access both NIDs 
on the immediate peer I am guessing it is something in my config. I wanted to 
see if a second set of eyes could point out what could I be doing wrong. Any 
idea?



Server is @ lustre-2.12.5-1.el7.x86_64 on CentOS7.8;

Lnet router is @ lustre-client-2.12.5-1.el7.x86_64 on CentOS7.8



Client is @lustre-client-2.14.0 on Ubuntu 22.04***;



Server(10.212.14.9@o2ib)

Lnet router(10.212.1.11@o2ib & 
10.215.25.76@o2ib2)

Client (10.215.25.74@o2ib2)



** Server: **

# lnetctl net show

net:

- net type: lo

  local NI(s):

- nid: 0@lo

  status: up

- net type: o2ib

  local NI(s):

- nid: 10.212.14.9@o2ib

  status: up

  interfaces:

  0: ib0



# lnetctl route show

route:

- net: o2ib1

  gateway: 10.212.15.16@o2ib

- net: o2ib1

  gateway: 10.212.16.20@o2ib

- net: o2ib2

  gateway: 10.212.1.11@o2ib



# lnetctl ping 10.212.1.11@o2ib

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.215.25.76@o2ib2

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.215.25.74@o2ib2

manage:

- ping:

  errno: -1

  descr: failed to ping 10.215.25.74@o2ib2: 
Input/output error



LNET router 

# lnetctl net show

net:

- net type: lo

  local NI(s):

- nid: 0@lo

  status: up

- net type: o2ib

  local NI(s):

- nid: 10.212.1.11@o2ib

  status: up

  interfaces:

  0: ib1

- net type: o2ib2

  local NI(s):

- nid: 10.215.25.76@o2ib2

  status: down

  interfaces:

  0: ib0





# lnetctl discover 10.212.1.11@o2ib

discover:

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl discover 10.215.25.76@o2ib2

discover:

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.212.14.9@o2ib

ping:

- primary nid: 10.212.14.9@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.14.9@o2ib



# lnetctl ping 10.215.25.75@o2ib2

ping:

- primary nid: 10.215.25.75@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.215.25.75@o2ib2



# lnetctl peer show

peer:

- primary nid: 10.215.25.74@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.215.25.74@o2ib2

  state: up

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  

Re: [lustre-discuss] Lnet config serving multiple routers and clients

[Horn, Chris via lustre-discuss]

Do you have the route to o2ib via 10.215.25.76@o2ib2 defined on the client?

Chris Horn

From: lustre-discuss  on behalf of 
Kumar, Amit via lustre-discuss 
Date: Wednesday, April 5, 2023 at 12:28 PM
To: lustre-discuss@lists.lustre.org 
Subject: [lustre-discuss] Lnet config serving multiple routers and clients

Dear Lustre team,



Below lustre server(showing one of many) is already serving current file system 
via lnet routers over o2ib1 to another cluster;



Now we are adding a new router to serve another new cluster and its clients 
over o2ib2;



Apparently, I can communicate via ping to lnet router’s both o2ib and o2ib2 
NIDs; Likewise, from client I can ping both o2ib and o2ib2 NIDs on lnet router. 
But end to end communication between client and server cannot find route to 
each other.



Initially I thought it me be related to LU-11641, given I can access both NIDs 
on the immediate peer I am guessing it is something in my config. I wanted to 
see if a second set of eyes could point out what could I be doing wrong. Any 
idea?



Server is @ lustre-2.12.5-1.el7.x86_64 on CentOS7.8;

Lnet router is @ lustre-client-2.12.5-1.el7.x86_64 on CentOS7.8



Client is @lustre-client-2.14.0 on Ubuntu 22.04***;



Server(10.212.14.9@o2ib)

Lnet router(10.212.1.11@o2ib & 10.215.25.76@o2ib2)

Client (10.215.25.74@o2ib2)



** Server: **

# lnetctl net show

net:

- net type: lo

  local NI(s):

- nid: 0@lo

  status: up

- net type: o2ib

  local NI(s):

- nid: 10.212.14.9@o2ib

  status: up

  interfaces:

  0: ib0



# lnetctl route show

route:

- net: o2ib1

  gateway: 10.212.15.16@o2ib

- net: o2ib1

  gateway: 10.212.16.20@o2ib

- net: o2ib2

  gateway: 10.212.1.11@o2ib



# lnetctl ping 10.212.1.11@o2ib

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.215.25.76@o2ib2

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.215.25.74@o2ib2

manage:

- ping:

  errno: -1

  descr: failed to ping 10.215.25.74@o2ib2: Input/output error



LNET router 

# lnetctl net show

net:

- net type: lo

  local NI(s):

- nid: 0@lo

  status: up

- net type: o2ib

  local NI(s):

- nid: 10.212.1.11@o2ib

  status: up

  interfaces:

  0: ib1

- net type: o2ib2

  local NI(s):

- nid: 10.215.25.76@o2ib2

  status: down

  interfaces:

  0: ib0





# lnetctl discover 10.212.1.11@o2ib

discover:

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl discover 10.215.25.76@o2ib2

discover:

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.212.14.9@o2ib

ping:

- primary nid: 10.212.14.9@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.14.9@o2ib



# lnetctl ping 10.215.25.75@o2ib2

ping:

- primary nid: 10.215.25.75@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.215.25.75@o2ib2



# lnetctl peer show

peer:

- primary nid: 10.215.25.74@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.215.25.74@o2ib2

  state: up

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

  state: up

- nid: 10.215.25.76@o2ib2

  state: up

- primary nid: 10.215.25.75@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.215.25.75@o2ib2

  state: up

- primary nid: 10.212.14.9@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.14.9@o2ib

  state: up



* Client *

# lnetctl net show

net:

- net type: lo

  local NI(s):

- nid: 0@lo

  status: up

- net type: o2ib2

  local NI(s):

- nid: 10.215.25.74@o2ib2

  status: up

  interfaces:

  0: ibp129s0f1





# lnetctl discover 10.212.1.11@o2ib

discover:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.215.25.76@o2ib2

- nid: 10.212.1.11@o2ib





# lnetctl ping 10.215.25.76@o2ib2

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2





# lnetctl ping 10.212.1.11@o2ib

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib

[lustre-discuss] Lnet config serving multiple routers and clients

[Kumar, Amit via lustre-discuss]

Dear Lustre team,



Below lustre server(showing one of many) is already serving current file system 
via lnet routers over o2ib1 to another cluster;



Now we are adding a new router to serve another new cluster and its clients 
over o2ib2;



Apparently, I can communicate via ping to lnet router's both o2ib and o2ib2 
NIDs; Likewise, from client I can ping both o2ib and o2ib2 NIDs on lnet router. 
But end to end communication between client and server cannot find route to 
each other.



Initially I thought it me be related to LU-11641, given I can access both NIDs 
on the immediate peer I am guessing it is something in my config. I wanted to 
see if a second set of eyes could point out what could I be doing wrong. Any 
idea?



Server is @ lustre-2.12.5-1.el7.x86_64 on CentOS7.8;

Lnet router is @ lustre-client-2.12.5-1.el7.x86_64 on CentOS7.8



Client is @lustre-client-2.14.0 on Ubuntu 22.04***;



Server(10.212.14.9@o2ib)

Lnet router(10.212.1.11@o2ib & 10.215.25.76@o2ib2)

Client (10.215.25.74@o2ib2)



** Server: **

# lnetctl net show

net:

- net type: lo

  local NI(s):

- nid: 0@lo

  status: up

- net type: o2ib

  local NI(s):

- nid: 10.212.14.9@o2ib

  status: up

  interfaces:

  0: ib0



# lnetctl route show

route:

- net: o2ib1

  gateway: 10.212.15.16@o2ib

- net: o2ib1

  gateway: 10.212.16.20@o2ib

- net: o2ib2

  gateway: 10.212.1.11@o2ib



# lnetctl ping 10.212.1.11@o2ib

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.215.25.76@o2ib2

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.215.25.74@o2ib2

manage:

- ping:

  errno: -1

  descr: failed to ping 10.215.25.74@o2ib2: Input/output error



LNET router 

# lnetctl net show

net:

- net type: lo

  local NI(s):

- nid: 0@lo

  status: up

- net type: o2ib

  local NI(s):

- nid: 10.212.1.11@o2ib

  status: up

  interfaces:

  0: ib1

- net type: o2ib2

  local NI(s):

- nid: 10.215.25.76@o2ib2

  status: down

  interfaces:

  0: ib0





# lnetctl discover 10.212.1.11@o2ib

discover:

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl discover 10.215.25.76@o2ib2

discover:

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.212.14.9@o2ib

ping:

- primary nid: 10.212.14.9@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.14.9@o2ib



# lnetctl ping 10.215.25.75@o2ib2

ping:

- primary nid: 10.215.25.75@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.215.25.75@o2ib2



# lnetctl peer show

peer:

- primary nid: 10.215.25.74@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.215.25.74@o2ib2

  state: up

- primary nid: 10.212.1.11@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

  state: up

- nid: 10.215.25.76@o2ib2

  state: up

- primary nid: 10.215.25.75@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.215.25.75@o2ib2

  state: up

- primary nid: 10.212.14.9@o2ib

  Multi-Rail: True

  peer ni:

- nid: 10.212.14.9@o2ib

  state: up



* Client *

# lnetctl net show

net:

- net type: lo

  local NI(s):

- nid: 0@lo

  status: up

- net type: o2ib2

  local NI(s):

- nid: 10.215.25.74@o2ib2

  status: up

  interfaces:

  0: ibp129s0f1





# lnetctl discover 10.212.1.11@o2ib

discover:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.215.25.76@o2ib2

- nid: 10.212.1.11@o2ib





# lnetctl ping 10.215.25.76@o2ib2

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2





# lnetctl ping 10.212.1.11@o2ib

ping:

- primary nid: 10.215.25.76@o2ib2

  Multi-Rail: True

  peer ni:

- nid: 10.212.1.11@o2ib

- nid: 10.215.25.76@o2ib2



# lnetctl ping 10.212.14.9@o2ib

manage:

- ping:

  errno: -1

  descr: failed to ping 10.212.14.9@o2ib: Input/output error



root@lnet1:~# lnetctl discover 10.212.14.9@o2ib

manage:

- discover:

  errno: -1

  descr: failed to discover 10.212.14.9@o2ib: No route t