Re: [lustre-discuss] OSS on compute node
I'm certainly not Andreas. That said... You're running an MPI simulation, assumingly across most or all of your 34 compute nodes. Lustre server operations, their lnet activity and the backend storage I/O will create a profound imbalance on the few compute nodes you designate to do both server and client operation. That and you expose yourself to deadlocks and other potentials mentioned earlier. I do not know how performant your login server is, but depending on the file operations of your simulations you could cavitate your login server. Also, you generally don't want users logging in on a node as critical as an MDS. You would be better served by allocating two of your compute nodes to just be Lustre servers, one mds/oss, the other an oss and run 32 clean client nodes. More stable, clean and in the end probably more workflow productivity over time. Fewer technical incidents. Just my opinion...others may differ. --Jeff On Fri, Oct 13, 2023 at 12:43 PM Fedele Stabile < fedele.stab...@fis.unical.it> wrote: > I believe in Linux is possible to limit the memory used by a user and also > it is possible to limit the amount of cpu used so I can limit resources for > group user and also if i put oss server in a vm i suppose i can limit cpu > and memory usage. > My scenario is: i have 34 compute nodes 512 GB RAM and 34 HD 16 TB each > that I can arrange in 9 nodes, i have also a management node that can be > used for LUSTRE metadata server, infiniband is 200 Gb/s > We make mhd simulations. > What Lustre configuration do you suggest? > > -- > *Da:* Andreas Dilger > *Inviato:* Venerdì, Ottobre 13, 2023 7:19:11 PM > *A:* Fedele Stabile > *Cc:* lustre-discuss@lists.lustre.org > *Oggetto:* Re: [lustre-discuss] OSS on compute node > > On Oct 13, 2023, at 20:58, Fedele Stabile > wrote: > > > Hello everyone, > We are in progress to integrate Lustre on our little HPC Cluster and we > would like to know if it is possible to use the same node in a cluster to > act as an OSS with disks and to also use it as a Compute Node and then > install a Lustre Client. > I know that the OSS server require a modified kernel so I suppose it can > be installed in a virtual machine using kvm on a compute node. > > > There isn't really a problem with running a client + OSS on the same node > anymore, nor is there a problem with an OSS running inside a VM (if you > have SR-IOV and enough CPU+RAM to run the server). > > *HOWEVER*, I don't think it would be good to have the client mounted on > the *VM host*, and then run the OSS on a *VM guest*. That could lead to > deadlocks and priority inversion if the client becomes busy, but depends on > the local OSS to flush dirty data from RAM and the OSS cannot run in the VM > because it doesn't have any RAM... > > If the client and OSS are BOTH run in VMs, or neither run in VMs, or only > the client run in a VM, then that should be OK, but may have reduced > performance due to the server contending with the client application. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Whamcloud > > > > > > > > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > -- -- Jeff Johnson Co-Founder Aeon Computing jeff.john...@aeoncomputing.com www.aeoncomputing.com t: 858-412-3810 x1001 f: 858-412-3845 m: 619-204-9061 4170 Morena Boulevard, Suite C - San Diego, CA 92117 High-Performance Computing / Lustre Filesystems / Scale-out Storage ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OSS on compute node
As I can remember, the BeeGFS is free, but some features are NOT, like the BeeOND. I did some detailed research years ago for my project. Best, Feng On Fri, Oct 13, 2023 at 3:54 PM Feng Zhang wrote: > > Yes. I do not have a chance to use BeeGFS, while I know some > institutes use it.https://io500.org/ > > Best, > > Feng > > On Fri, Oct 13, 2023 at 3:49 PM Fedele Stabile > wrote: > > > > I have to study in detail beegfs, > > Is it usable on a little HPC Cluster? > > > > > > From: Feng Zhang > > Sent: Friday, October 13, 2023 9:07:48 PM > > To: Fedele Stabile > > Cc: lustre-discuss@lists.lustre.org > > Subject: Re: [lustre-discuss] OSS on compute node > > > > In theory it should work, but may have some performance issues though. > > > > This idea is also similar to BeeGFS BeeOND (BeeGFS on-demand). > > > > Should run computer(client) nodes in VM, or container, to avoid any > > users' app to crash the whole node intentionally or not. > > > > Best, > > > > Feng > > > > On Fri, Oct 13, 2023 at 7:00 AM Fedele Stabile > > wrote: > > > > > > Hello everyone, > > > We are in progress to integrate Lustre on our little HPC Cluster and we > > > would like to know if it is possible to use the same node in a cluster to > > > act as an OSS with disks and to also use it as a Compute Node and then > > > install a Lustre Client. > > > I know that the OSS server require a modified kernel so I suppose it can > > > be installed in a virtual machine using kvm on a compute node. > > > > > > Greetings from UniCal Cosenza-Calabria Italy, > > > Fedele > > > ___ > > > lustre-discuss mailing list > > > lustre-discuss@lists.lustre.org > > > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OSS on compute node
Yes. I do not have a chance to use BeeGFS, while I know some institutes use it.https://io500.org/ Best, Feng On Fri, Oct 13, 2023 at 3:49 PM Fedele Stabile wrote: > > I have to study in detail beegfs, > Is it usable on a little HPC Cluster? > > > From: Feng Zhang > Sent: Friday, October 13, 2023 9:07:48 PM > To: Fedele Stabile > Cc: lustre-discuss@lists.lustre.org > Subject: Re: [lustre-discuss] OSS on compute node > > In theory it should work, but may have some performance issues though. > > This idea is also similar to BeeGFS BeeOND (BeeGFS on-demand). > > Should run computer(client) nodes in VM, or container, to avoid any > users' app to crash the whole node intentionally or not. > > Best, > > Feng > > On Fri, Oct 13, 2023 at 7:00 AM Fedele Stabile > wrote: > > > > Hello everyone, > > We are in progress to integrate Lustre on our little HPC Cluster and we > > would like to know if it is possible to use the same node in a cluster to > > act as an OSS with disks and to also use it as a Compute Node and then > > install a Lustre Client. > > I know that the OSS server require a modified kernel so I suppose it can be > > installed in a virtual machine using kvm on a compute node. > > > > Greetings from UniCal Cosenza-Calabria Italy, > > Fedele > > ___ > > lustre-discuss mailing list > > lustre-discuss@lists.lustre.org > > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OSS on compute node
I have to study in detail beegfs, Is it usable on a little HPC Cluster? From: Feng Zhang Sent: Friday, October 13, 2023 9:07:48 PM To: Fedele Stabile Cc: lustre-discuss@lists.lustre.org Subject: Re: [lustre-discuss] OSS on compute node In theory it should work, but may have some performance issues though. This idea is also similar to BeeGFS BeeOND (BeeGFS on-demand). Should run computer(client) nodes in VM, or container, to avoid any users' app to crash the whole node intentionally or not. Best, Feng On Fri, Oct 13, 2023 at 7:00 AM Fedele Stabile wrote: > > Hello everyone, > We are in progress to integrate Lustre on our little HPC Cluster and we would > like to know if it is possible to use the same node in a cluster to act as an > OSS with disks and to also use it as a Compute Node and then install a Lustre > Client. > I know that the OSS server require a modified kernel so I suppose it can be > installed in a virtual machine using kvm on a compute node. > > Greetings from UniCal Cosenza-Calabria Italy, > Fedele > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OSS on compute node
I believe in Linux is possible to limit the memory used by a user and also it is possible to limit the amount of cpu used so I can limit resources for group user and also if i put oss server in a vm i suppose i can limit cpu and memory usage. My scenario is: i have 34 compute nodes 512 GB RAM and 34 HD 16 TB each that I can arrange in 9 nodes, i have also a management node that can be used for LUSTRE metadata server, infiniband is 200 Gb/s We make mhd simulations. What Lustre configuration do you suggest? Da: Andreas Dilger Inviato: Venerdì, Ottobre 13, 2023 7:19:11 PM A: Fedele Stabile Cc: lustre-discuss@lists.lustre.org Oggetto: Re: [lustre-discuss] OSS on compute node On Oct 13, 2023, at 20:58, Fedele Stabile mailto:fedele.stab...@fis.unical.it>> wrote: Hello everyone, We are in progress to integrate Lustre on our little HPC Cluster and we would like to know if it is possible to use the same node in a cluster to act as an OSS with disks and to also use it as a Compute Node and then install a Lustre Client. I know that the OSS server require a modified kernel so I suppose it can be installed in a virtual machine using kvm on a compute node. There isn't really a problem with running a client + OSS on the same node anymore, nor is there a problem with an OSS running inside a VM (if you have SR-IOV and enough CPU+RAM to run the server). *HOWEVER*, I don't think it would be good to have the client mounted on the *VM host*, and then run the OSS on a *VM guest*. That could lead to deadlocks and priority inversion if the client becomes busy, but depends on the local OSS to flush dirty data from RAM and the OSS cannot run in the VM because it doesn't have any RAM... If the client and OSS are BOTH run in VMs, or neither run in VMs, or only the client run in a VM, then that should be OK, but may have reduced performance due to the server contending with the client application. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OSS on compute node
In theory it should work, but may have some performance issues though. This idea is also similar to BeeGFS BeeOND (BeeGFS on-demand). Should run computer(client) nodes in VM, or container, to avoid any users' app to crash the whole node intentionally or not. Best, Feng On Fri, Oct 13, 2023 at 7:00 AM Fedele Stabile wrote: > > Hello everyone, > We are in progress to integrate Lustre on our little HPC Cluster and we would > like to know if it is possible to use the same node in a cluster to act as an > OSS with disks and to also use it as a Compute Node and then install a Lustre > Client. > I know that the OSS server require a modified kernel so I suppose it can be > installed in a virtual machine using kvm on a compute node. > > Greetings from UniCal Cosenza-Calabria Italy, > Fedele > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OSS on compute node
Skydiving with an anvil is *possible* ...but not advisable. --Jeff On Fri, Oct 13, 2023 at 10:21 AM Andreas Dilger via lustre-discuss < lustre-discuss@lists.lustre.org> wrote: > On Oct 13, 2023, at 20:58, Fedele Stabile > wrote: > > > Hello everyone, > We are in progress to integrate Lustre on our little HPC Cluster and we > would like to know if it is possible to use the same node in a cluster to > act as an OSS with disks and to also use it as a Compute Node and then > install a Lustre Client. > I know that the OSS server require a modified kernel so I suppose it can > be installed in a virtual machine using kvm on a compute node. > > > There isn't really a problem with running a client + OSS on the same node > anymore, nor is there a problem with an OSS running inside a VM (if you > have SR-IOV and enough CPU+RAM to run the server). > > *HOWEVER*, I don't think it would be good to have the client mounted on > the *VM host*, and then run the OSS on a *VM guest*. That could lead to > deadlocks and priority inversion if the client becomes busy, but depends on > the local OSS to flush dirty data from RAM and the OSS cannot run in the VM > because it doesn't have any RAM... > > If the client and OSS are BOTH run in VMs, or neither run in VMs, or only > the client run in a VM, then that should be OK, but may have reduced > performance due to the server contending with the client application. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Whamcloud > > > > > > > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > -- -- Jeff Johnson Co-Founder Aeon Computing jeff.john...@aeoncomputing.com www.aeoncomputing.com t: 858-412-3810 x1001 f: 858-412-3845 m: 619-204-9061 4170 Morena Boulevard, Suite C - San Diego, CA 92117 High-Performance Computing / Lustre Filesystems / Scale-out Storage ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OSS on compute node
On Oct 13, 2023, at 20:58, Fedele Stabile mailto:fedele.stab...@fis.unical.it>> wrote: Hello everyone, We are in progress to integrate Lustre on our little HPC Cluster and we would like to know if it is possible to use the same node in a cluster to act as an OSS with disks and to also use it as a Compute Node and then install a Lustre Client. I know that the OSS server require a modified kernel so I suppose it can be installed in a virtual machine using kvm on a compute node. There isn't really a problem with running a client + OSS on the same node anymore, nor is there a problem with an OSS running inside a VM (if you have SR-IOV and enough CPU+RAM to run the server). *HOWEVER*, I don't think it would be good to have the client mounted on the *VM host*, and then run the OSS on a *VM guest*. That could lead to deadlocks and priority inversion if the client becomes busy, but depends on the local OSS to flush dirty data from RAM and the OSS cannot run in the VM because it doesn't have any RAM... If the client and OSS are BOTH run in VMs, or neither run in VMs, or only the client run in a VM, then that should be OK, but may have reduced performance due to the server contending with the client application. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OSS on compute node
> What is the resource consumption (memory anc CPU) of a storage server? The impression I've been under is that it's less about being able to put sufficient resources in one host and more about the potential for deadlock. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] OSS on compute node
Hello everyone, We are in progress to integrate Lustre on our little HPC Cluster and we would like to know if it is possible to use the same node in a cluster to act as an OSS with disks and to also use it as a Compute Node and then install a Lustre Client. I know that the OSS server require a modified kernel so I suppose it can be installed in a virtual machine using kvm on a compute node. Greetings from UniCal Cosenza-Calabria Italy, Fedele ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org