Re: [lustre-discuss] OSS on compute node

2023-10-13 Thread Jeff Johnson 
I'm certainly not Andreas. That said...

You're running an MPI simulation, assumingly across most or all of your 34
compute nodes. Lustre server operations, their lnet activity and the
backend storage I/O will create a profound imbalance on the few compute
nodes you designate to do both server and client operation. That and you
expose yourself to deadlocks and other potentials mentioned earlier. I do
not know how performant your login server is, but depending on the file
operations of your simulations you could cavitate your login server. Also,
you generally don't want users logging in on a node as critical as an MDS.

You would be better served by allocating two of your compute nodes to just
be Lustre servers, one mds/oss, the other an oss and run 32 clean client
nodes. More stable, clean and in the end  probably more workflow
productivity over time. Fewer technical incidents.

Just my opinion...others may differ.

--Jeff



On Fri, Oct 13, 2023 at 12:43 PM Fedele Stabile <
fedele.stab...@fis.unical.it> wrote:

> I believe in Linux is possible to limit the memory used by a user and also
> it is possible to limit the amount of cpu used so I can limit resources for
> group user and also if i put oss server in a vm i suppose i can limit cpu
> and memory usage.
> My scenario is: i have 34 compute nodes 512 GB RAM and 34 HD 16 TB each
> that I can arrange in 9 nodes, i have also a management node that can be
> used for LUSTRE metadata server, infiniband is 200 Gb/s
> We make mhd simulations.
> What Lustre configuration do you suggest?
>
> --
> *Da:* Andreas Dilger 
> *Inviato:* Venerdì, Ottobre 13, 2023 7:19:11 PM
> *A:* Fedele Stabile 
> *Cc:* lustre-discuss@lists.lustre.org 
> *Oggetto:* Re: [lustre-discuss] OSS on compute node
>
> On Oct 13, 2023, at 20:58, Fedele Stabile 
> wrote:
>
>
> Hello everyone,
> We are in progress to integrate Lustre on our little HPC Cluster and we
> would like to know if it is possible to use the same node in a cluster to
> act as an OSS with disks and to also use it as a Compute Node and then
> install a Lustre Client.
> I know that the OSS server require a modified kernel so I suppose it can
> be installed in a virtual machine using kvm on a compute node.
>
>
> There isn't really a problem with running a client + OSS on the same node
> anymore, nor is there a problem with an OSS running inside a VM (if you
> have SR-IOV and enough CPU+RAM to run the server).
>
> *HOWEVER*, I don't think it would be good to have the client mounted on
> the *VM host*, and then run the OSS on a *VM guest*.  That could lead to
> deadlocks and priority inversion if the client becomes busy, but depends on
> the local OSS to flush dirty data from RAM and the OSS cannot run in the VM
> because it doesn't have any RAM...
>
> If the client and OSS are BOTH run in VMs, or neither run in VMs, or only
> the client run in a VM, then that should be OK, but may have reduced
> performance due to the server contending with the client application.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Lustre Principal Architect
> Whamcloud
>
>
>
>
>
>
>
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>


-- 
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite C - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OSS on compute node

2023-10-13 Thread Feng Zhang via lustre-discuss 
As I can remember, the BeeGFS is free, but some features are NOT, like
the BeeOND. I did some detailed research years ago for my project.

Best,

Feng

On Fri, Oct 13, 2023 at 3:54 PM Feng Zhang  wrote:
>
> Yes. I do not have a chance to use BeeGFS, while I know some
> institutes use it.https://io500.org/
>
> Best,
>
> Feng
>
> On Fri, Oct 13, 2023 at 3:49 PM Fedele Stabile
>  wrote:
> >
> > I have to study in detail beegfs,
> >  Is it usable on a little HPC Cluster?
> >
> > 
> > From: Feng Zhang 
> > Sent: Friday, October 13, 2023 9:07:48 PM
> > To: Fedele Stabile 
> > Cc: lustre-discuss@lists.lustre.org 
> > Subject: Re: [lustre-discuss] OSS on compute node
> >
> > In theory it should work, but may have some performance issues though.
> >
> > This idea is also similar to BeeGFS BeeOND (BeeGFS on-demand).
> >
> > Should run computer(client) nodes in VM, or container, to avoid any
> > users' app to crash the whole node intentionally or not.
> >
> > Best,
> >
> > Feng
> >
> > On Fri, Oct 13, 2023 at 7:00 AM Fedele Stabile
> >  wrote:
> > >
> > > Hello everyone,
> > > We are in progress to integrate Lustre on our little HPC Cluster and we 
> > > would like to know if it is possible to use the same node in a cluster to 
> > > act as an OSS with disks and to also use it as a Compute Node and then 
> > > install a Lustre Client.
> > > I know that the OSS server require a modified kernel so I suppose it can 
> > > be installed in a virtual machine using kvm on a compute node.
> > >
> > > Greetings from UniCal Cosenza-Calabria Italy,
> > > Fedele
> > > ___
> > > lustre-discuss mailing list
> > > lustre-discuss@lists.lustre.org
> > > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OSS on compute node

2023-10-13 Thread Feng Zhang via lustre-discuss 
Yes. I do not have a chance to use BeeGFS, while I know some
institutes use it.https://io500.org/

Best,

Feng

On Fri, Oct 13, 2023 at 3:49 PM Fedele Stabile
 wrote:
>
> I have to study in detail beegfs,
>  Is it usable on a little HPC Cluster?
>
> 
> From: Feng Zhang 
> Sent: Friday, October 13, 2023 9:07:48 PM
> To: Fedele Stabile 
> Cc: lustre-discuss@lists.lustre.org 
> Subject: Re: [lustre-discuss] OSS on compute node
>
> In theory it should work, but may have some performance issues though.
>
> This idea is also similar to BeeGFS BeeOND (BeeGFS on-demand).
>
> Should run computer(client) nodes in VM, or container, to avoid any
> users' app to crash the whole node intentionally or not.
>
> Best,
>
> Feng
>
> On Fri, Oct 13, 2023 at 7:00 AM Fedele Stabile
>  wrote:
> >
> > Hello everyone,
> > We are in progress to integrate Lustre on our little HPC Cluster and we 
> > would like to know if it is possible to use the same node in a cluster to 
> > act as an OSS with disks and to also use it as a Compute Node and then 
> > install a Lustre Client.
> > I know that the OSS server require a modified kernel so I suppose it can be 
> > installed in a virtual machine using kvm on a compute node.
> >
> > Greetings from UniCal Cosenza-Calabria Italy,
> > Fedele
> > ___
> > lustre-discuss mailing list
> > lustre-discuss@lists.lustre.org
> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OSS on compute node

2023-10-13 Thread Fedele Stabile 
I have to study in detail beegfs,
 Is it usable on a little HPC Cluster?


From: Feng Zhang 
Sent: Friday, October 13, 2023 9:07:48 PM
To: Fedele Stabile 
Cc: lustre-discuss@lists.lustre.org 
Subject: Re: [lustre-discuss] OSS on compute node

In theory it should work, but may have some performance issues though.

This idea is also similar to BeeGFS BeeOND (BeeGFS on-demand).

Should run computer(client) nodes in VM, or container, to avoid any
users' app to crash the whole node intentionally or not.

Best,

Feng

On Fri, Oct 13, 2023 at 7:00 AM Fedele Stabile
 wrote:
>
> Hello everyone,
> We are in progress to integrate Lustre on our little HPC Cluster and we would 
> like to know if it is possible to use the same node in a cluster to act as an 
> OSS with disks and to also use it as a Compute Node and then install a Lustre 
> Client.
> I know that the OSS server require a modified kernel so I suppose it can be 
> installed in a virtual machine using kvm on a compute node.
>
> Greetings from UniCal Cosenza-Calabria Italy,
> Fedele
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OSS on compute node

2023-10-13 Thread Fedele Stabile 
I believe in Linux is possible to limit the memory used by a user and also it 
is possible to limit the amount of cpu used so I can limit resources for group 
user and also if i put oss server in a vm i suppose i can limit cpu and memory 
usage.
My scenario is: i have 34 compute nodes 512 GB RAM and 34 HD 16 TB each that I 
can arrange in 9 nodes, i have also a management node that can be used for 
LUSTRE metadata server, infiniband is 200 Gb/s
We make mhd simulations.
What Lustre configuration do you suggest?


Da: Andreas Dilger 
Inviato: Venerdì, Ottobre 13, 2023 7:19:11 PM
A: Fedele Stabile 
Cc: lustre-discuss@lists.lustre.org 
Oggetto: Re: [lustre-discuss] OSS on compute node

On Oct 13, 2023, at 20:58, Fedele Stabile 
mailto:fedele.stab...@fis.unical.it>> wrote:

Hello everyone,
We are in progress to integrate Lustre on our little HPC Cluster and we would 
like to know if it is possible to use the same node in a cluster to act as an 
OSS with disks and to also use it as a Compute Node and then install a Lustre 
Client.
I know that the OSS server require a modified kernel so I suppose it can be 
installed in a virtual machine using kvm on a compute node.

There isn't really a problem with running a client + OSS on the same node 
anymore, nor is there a problem with an OSS running inside a VM (if you have 
SR-IOV and enough CPU+RAM to run the server).

*HOWEVER*, I don't think it would be good to have the client mounted on the *VM 
host*, and then run the OSS on a *VM guest*.  That could lead to deadlocks and 
priority inversion if the client becomes busy, but depends on the local OSS to 
flush dirty data from RAM and the OSS cannot run in the VM because it doesn't 
have any RAM...

If the client and OSS are BOTH run in VMs, or neither run in VMs, or only the 
client run in a VM, then that should be OK, but may have reduced performance 
due to the server contending with the client application.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud








___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OSS on compute node

2023-10-13 Thread Feng Zhang via lustre-discuss 
In theory it should work, but may have some performance issues though.

This idea is also similar to BeeGFS BeeOND (BeeGFS on-demand).

Should run computer(client) nodes in VM, or container, to avoid any
users' app to crash the whole node intentionally or not.

Best,

Feng

On Fri, Oct 13, 2023 at 7:00 AM Fedele Stabile
 wrote:
>
> Hello everyone,
> We are in progress to integrate Lustre on our little HPC Cluster and we would 
> like to know if it is possible to use the same node in a cluster to act as an 
> OSS with disks and to also use it as a Compute Node and then install a Lustre 
> Client.
> I know that the OSS server require a modified kernel so I suppose it can be 
> installed in a virtual machine using kvm on a compute node.
>
> Greetings from UniCal Cosenza-Calabria Italy,
> Fedele
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OSS on compute node

2023-10-13 Thread Jeff Johnson 
Skydiving with an anvil is *possible* ...but not advisable.

--Jeff


On Fri, Oct 13, 2023 at 10:21 AM Andreas Dilger via lustre-discuss <
lustre-discuss@lists.lustre.org> wrote:

> On Oct 13, 2023, at 20:58, Fedele Stabile 
> wrote:
>
>
> Hello everyone,
> We are in progress to integrate Lustre on our little HPC Cluster and we
> would like to know if it is possible to use the same node in a cluster to
> act as an OSS with disks and to also use it as a Compute Node and then
> install a Lustre Client.
> I know that the OSS server require a modified kernel so I suppose it can
> be installed in a virtual machine using kvm on a compute node.
>
>
> There isn't really a problem with running a client + OSS on the same node
> anymore, nor is there a problem with an OSS running inside a VM (if you
> have SR-IOV and enough CPU+RAM to run the server).
>
> *HOWEVER*, I don't think it would be good to have the client mounted on
> the *VM host*, and then run the OSS on a *VM guest*.  That could lead to
> deadlocks and priority inversion if the client becomes busy, but depends on
> the local OSS to flush dirty data from RAM and the OSS cannot run in the VM
> because it doesn't have any RAM...
>
> If the client and OSS are BOTH run in VMs, or neither run in VMs, or only
> the client run in a VM, then that should be OK, but may have reduced
> performance due to the server contending with the client application.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Lustre Principal Architect
> Whamcloud
>
>
>
>
>
>
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>


-- 
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite C - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OSS on compute node

2023-10-13 Thread Andreas Dilger via lustre-discuss 
On Oct 13, 2023, at 20:58, Fedele Stabile 
mailto:fedele.stab...@fis.unical.it>> wrote:

Hello everyone,
We are in progress to integrate Lustre on our little HPC Cluster and we would 
like to know if it is possible to use the same node in a cluster to act as an 
OSS with disks and to also use it as a Compute Node and then install a Lustre 
Client.
I know that the OSS server require a modified kernel so I suppose it can be 
installed in a virtual machine using kvm on a compute node.

There isn't really a problem with running a client + OSS on the same node 
anymore, nor is there a problem with an OSS running inside a VM (if you have 
SR-IOV and enough CPU+RAM to run the server).

*HOWEVER*, I don't think it would be good to have the client mounted on the *VM 
host*, and then run the OSS on a *VM guest*.  That could lead to deadlocks and 
priority inversion if the client becomes busy, but depends on the local OSS to 
flush dirty data from RAM and the OSS cannot run in the VM because it doesn't 
have any RAM...

If the client and OSS are BOTH run in VMs, or neither run in VMs, or only the 
client run in a VM, then that should be OK, but may have reduced performance 
due to the server contending with the client application.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud







___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OSS on compute node

2023-10-13 Thread Laura Hild via lustre-discuss 
> What is the resource consumption (memory anc CPU) of a storage server?

The impression I've been under is that it's less about being able to put 
sufficient resources in one host and more about the potential for deadlock.

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] R: OSS on compute node

2023-10-13 Thread Fedele Stabile 
Thank you for the answer,
KVM is able  to use SR-IOV? 
What is the resource consumption (memory anc CPU) of a storage server?
Fedele


Da: Walker Haddock 
Inviato: venerdì 13 ottobre 2023 13:54
A: Fedele Stabile
Oggetto: Re: [lustre-discuss] OSS on compute node

There will be great risks to your storage and compute when sharing a node this 
way. However, we do this in development sandboxes where we have infrastructure 
as code to redeploy quickly and effortlessly when bad things happen. You should 
consider putting your Lustre nodes in KVMs, use SR-IOV and consider the impact 
of losing data. The resources on the storage node remaining may result in lower 
computing power, especially memory.
ZFS might be a better choice for your volumes.

Regards,
Walker

On Fri, Oct 13, 2023 at 6:01 AM Fedele Stabile 
mailto:fedele.stab...@fis.unical.it>> wrote:
Hello everyone,
We are in progress to integrate Lustre on our little HPC Cluster and we would 
like to know if it is possible to use the same node in a cluster to act as an 
OSS with disks and to also use it as a Compute Node and then install a Lustre 
Client.
I know that the OSS server require a modified kernel so I suppose it can be 
installed in a virtual machine using kvm on a compute node.

Greetings from UniCal Cosenza-Calabria Italy,
Fedele
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] OSS on compute node

2023-10-13 Thread Fedele Stabile 
Hello everyone,
We are in progress to integrate Lustre on our little HPC Cluster and we would 
like to know if it is possible to use the same node in a cluster to act as an 
OSS with disks and to also use it as a Compute Node and then install a Lustre 
Client.
I know that the OSS server require a modified kernel so I suppose it can be 
installed in a virtual machine using kvm on a compute node.

Greetings from UniCal Cosenza-Calabria Italy,
Fedele
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org