Re: [lustre-discuss] Lustre as /home directory
Hello, actually, I have a different opinion to what the conclusion of this thread seems to be since I believe that Lustre is no bad choice for /home. We are using Lustre as /home since 2005 and had no bad experiences with it. The reason why we use Lustre for /home and for /scratch is that we do not want to support different file system and hardware solutions. Of course, the best solution always depends on your applications and your requirements. But we have 1000s of active users on more than 2500 nodes and they are using all kinds of applications. However, we also regularly educate our users to do the right thing, e.g. use other Lustre file systems, use local disks or fix their applications. BTW, Lustre jobstats are great for identifying misbehaving users. As mentioned, a good backup is important for /home for every kind of file system solution. With more than 100 million files more than 100 TB capacity doing disaster recovery in time is challenging. I've presented our solution for this at LAD'14, see https://www.eofs.eu/_media/events/lad14/06_roland_laifer_kit_lad14_20140922.pdf And we really needed it last year which was presented at the HPC-IODC, see https://wr.informatik.uni-hamburg.de/_media/events/2017/iodc-17-laifer.pdf Regards, Roland Am 16.02.2018 um 17:10 schrieb Carlson, Timothy S: > I'll just add +1 to this thread. /home on NFS for software builds, small > files, lots of metadata operations. Lustre for the rest. Users will do the > wrong thing even after education. > > Tim > > -Original Message- > From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On > Behalf Of Steve Barnet > Sent: Friday, February 16, 2018 7:21 AM > To: lustre-discuss@lists.lustre.org > Subject: Re: [lustre-discuss] Lustre as /home directory > > On 2/16/18 8:53 AM, Michael Watters wrote: >> Can't be much worse than NFS. > > > Oh yes, it can! It will really depend upon how your users work with it. If > you regularly have many people building their software chains, that can drag > your interactive response to its knees very quickly. > If your users also like a to drop many thousands of small files into just a > few directories, you will definitely notice that as well. > > We have found that NFS (while it certainly will also > suffer) generally holds up better than Lustre in those use cases. No > particularly earth shattering news there. > > We have generally kept /home on NFS, and have fairly restrictive quotas > there. The idea is that /home is used for software builds and final results, > but that the heavy cluster processing workloads are handled by lustre. > > This has worked out OK. We still run into plenty of cases where people do the > wrong thing, but we can generally redirect them to the right place. > > Best, > > ---Steve > > > >> >> >> On 02/15/2018 10:30 AM, Mark Hahn wrote: >>>> My question is, Is it advisable to have /home in Lustre since users >>>> data will be of small files (less than 5MB)? >>> >>> certainly it works, but is not very pleasant for metadata-intensive >>> activity, such as compiling. >>> ___ >>> lustre-discuss mailing list >>> lustre-discuss@lists.lustre.org >>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> >> ___ >> lustre-discuss mailing list >> lustre-discuss@lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Lustre as /home directory
I'll just add +1 to this thread. /home on NFS for software builds, small files, lots of metadata operations. Lustre for the rest. Users will do the wrong thing even after education. Tim -Original Message- From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Steve Barnet Sent: Friday, February 16, 2018 7:21 AM To: lustre-discuss@lists.lustre.org Subject: Re: [lustre-discuss] Lustre as /home directory On 2/16/18 8:53 AM, Michael Watters wrote: > Can't be much worse than NFS. Oh yes, it can! It will really depend upon how your users work with it. If you regularly have many people building their software chains, that can drag your interactive response to its knees very quickly. If your users also like a to drop many thousands of small files into just a few directories, you will definitely notice that as well. We have found that NFS (while it certainly will also suffer) generally holds up better than Lustre in those use cases. No particularly earth shattering news there. We have generally kept /home on NFS, and have fairly restrictive quotas there. The idea is that /home is used for software builds and final results, but that the heavy cluster processing workloads are handled by lustre. This has worked out OK. We still run into plenty of cases where people do the wrong thing, but we can generally redirect them to the right place. Best, ---Steve > > > On 02/15/2018 10:30 AM, Mark Hahn wrote: >>> My question is, Is it advisable to have /home in Lustre since users >>> data will be of small files (less than 5MB)? >> >> certainly it works, but is not very pleasant for metadata-intensive >> activity, such as compiling. >> ___ >> lustre-discuss mailing list >> lustre-discuss@lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Lustre as /home directory
On 2/16/18 8:53 AM, Michael Watters wrote: Can't be much worse than NFS. Oh yes, it can! It will really depend upon how your users work with it. If you regularly have many people building their software chains, that can drag your interactive response to its knees very quickly. If your users also like a to drop many thousands of small files into just a few directories, you will definitely notice that as well. We have found that NFS (while it certainly will also suffer) generally holds up better than Lustre in those use cases. No particularly earth shattering news there. We have generally kept /home on NFS, and have fairly restrictive quotas there. The idea is that /home is used for software builds and final results, but that the heavy cluster processing workloads are handled by lustre. This has worked out OK. We still run into plenty of cases where people do the wrong thing, but we can generally redirect them to the right place. Best, ---Steve On 02/15/2018 10:30 AM, Mark Hahn wrote: My question is, Is it advisable to have /home in Lustre since users data will be of small files (less than 5MB)? certainly it works, but is not very pleasant for metadata-intensive activity, such as compiling. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Lustre as /home directory
Can't be much worse than NFS. On 02/15/2018 10:30 AM, Mark Hahn wrote: >> My question is, Is it advisable to have /home in Lustre since users data >> will be of small files (less than 5MB)? > > certainly it works, but is not very pleasant for metadata-intensive > activity, such as compiling. > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Lustre as /home directory
What will your backup/snapshot strategy going to be and how will you implement it? On Thu, Feb 15, 2018 at 5:30 PM, Mark Hahn wrote: > My question is, Is it advisable to have /home in Lustre since users data >> will be of small files (less than 5MB)? >> > > certainly it works, but is not very pleasant for metadata-intensive > activity, such as compiling. > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Lustre as /home directory
My question is, Is it advisable to have /home in Lustre since users data will be of small files (less than 5MB)? certainly it works, but is not very pleasant for metadata-intensive activity, such as compiling. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Lustre as /home directory
Hi, I don't know if the experts think it is advisable or not, but it does work. I have been using /home on Lustre for years now, and not encountered any problems. gr, Martin From: "Ravi Konila" To: "Lustre Discuss" Date: 15-02-2018 08:25 Subject:[lustre-discuss] Lustre as /home directory Sent by:"lustre-discuss" Hi, We are have working Lustre 2.8 with 100TB with 50TB for scratch and 50TB for stage. Now our end users department wants to have /home in Lustre (presently home is available from NFS to all nodes). My question is, Is it advisable to have /home in Lustre since users data will be of small files (less than 5MB)? Regards Ravi Konila ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
