Re: [lustre-discuss] How to speed up Lustre
ned out to not be feasible for us. PFL has been a win for us here, for that reason. Our conclusion was that in order to take advantage of the performance improvements of DoM, you need enough money for lots of flash, or you need enough staff time to manage the DoM layouts to fit into that flash. We have neither of those conditions, and we find that using PFL and flash OST's for small files is working very well for us. Regards, Marion From: =?utf-8?B?VGFuZXIgS0FSQUfDlkw=?= mailto:kara...@aselsan.com.tr>> To: Marion Hakanson mailto:hakan...@ohsu.edu>> CC: "lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>" mailto:lustre-discuss@lists.lustre.org>> Date: Tue, 22 Feb 2022 04:53:03 + UNCLASSIFIED Thank you for sharing your experience. I was thinking that DoM is built in feature and it can be enabled/disabled online for a certain directories. What do you mean by reformat to converting to DoM (or away from it). I think just Metadata target size is important. I also thought creating flash OST on metadata server. But I was not sure what to install on metadata server for this purpose. Can Metadata server be an OSS server at the same time? If it is possible I would prefer flash OST on Metadata server instead of DoM. Because Our metadata target size is small, it seems I have to do risky operations to expand size. imho, because of the less RPC traffic DoM shows more performance than flash OST. Am I right? Best Regards; From: Marion Hakanson mailto:hakan...@ohsu.edu>> Sent: Thursday, February 17, 2022 8:20 PM To: Taner KARAGÖL mailto:kara...@aselsan.com.tr>> Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> Subject: Re: [lustre-discuss] How to speed up Lustre We started with DoM on our new Lustre system a couple years ago. - Converting to DoM (or away from it) is a full-reformat operation. - DoM uses a fixed amount of metadata space (64k minimum for us) for every file, even those smaller than 64k. Basically, DoM uses a lot of flash metadata space, more than we planned for, and more than we could afford. We ended up switching to a PFL arrangement, where the first 64k lives on flash OST's (mounted on our metadata servers), and the remainder of larger files lives on HDD OST's. This is working very well for our small-file workloads, and uses less flash space than the DoM configuration did. Since you don't already have DoM in effect, it may be possible that you could add flash OST's, configure a PFL, and then use "lfs migrate" to re-layout existing files into the new OST's. Your mileage may vary, so be safe! Regards, Marion On Feb 14, 2022, at 03:32, Taner KARAGÖL via lustre-discuss mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org>> wrote:  UNCLASSIFIED Hi Everybody; We have a performance problem with small files on our HPC system (120 compute nodes). Our all OSS targets are classic spinning HDDs. To speed up, I want to configure Data on Metadata. Our metadata target has SDD disks. Underlying file systems are ZFS (for OSS and Meta) Lustre version: 2.12.5 ZFS version: .0.7.13 Our Lustre file system size is 720TB (2 OSS servers, 1 enclosure with 6 zpools), Metadata file system size is 2.1TB(1 enclosure and 1 metadata target). What is your opinions to speed up this setup? I want to configure DoM but I am concerning about Metadata size. My questions: 1. How can I increase Medatadata size? Metadata enclosure has a empty slots. Is there a way to increase size online/offline? 2. Is it possible to migrate big files from DoM to OSS targets completely? Off course online migration. (So I think I can free Metadata for new small files). Best Regards; Taner Dikkat: Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir. Attention: This e-mail message is privileged and confidential. If you are not the intended recipient please delete the message and notify the sender. E-mails to and from the company are monitored for operational reasons and in accordance with lawful business practices. Any views or opinions presented are solely those of the author and do not necessarily represent the views of the company. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org> https://ur
Re: [lustre-discuss] How to speed up Lustre
T on Metadata server instead of DoM. Because Our metadata target size is small, it seems I have to do risky operations to expand size. Yes, our metadata servers are also OSS's at the same time. The flash OST's are separate volumes (and drives) from the MDT's, so less scary (:-). kara...@aselsan.com.tr<mailto:kara...@aselsan.com.tr><mailto:kara...@aselsan.com.tr> said: imho, because of the less RPC traffic DoM shows more performance than flash OST. Am I right? The documentation does say there that using DoM for small files will produce less RPC traffic than using OST's for small files. But as I said earlier, for us, the amount of flash needed to support DoM was a lot higher than with the flash OST approach (we have a high percentage, by number, of small files). I'll also note that we had a wish to mostly "set and forget" the layout for our Lustre filesystem. We have not figured out a way to predict or control where small files (or large ones) are going to end up, so trying to craft optimal layouts in particular directories for particular file sizes has turned out to not be feasible for us. PFL has been a win for us here, for that reason. Our conclusion was that in order to take advantage of the performance improvements of DoM, you need enough money for lots of flash, or you need enough staff time to manage the DoM layouts to fit into that flash. We have neither of those conditions, and we find that using PFL and flash OST's for small files is working very well for us. Regards, Marion From: =?utf-8?B?VGFuZXIgS0FSQUfDlkw=?= mailto:kara...@aselsan.com.tr><mailto:kara...@aselsan.com.tr>> To: Marion Hakanson mailto:hakan...@ohsu.edu><mailto:hakan...@ohsu.edu>> CC: "lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org>" mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org>> Date: Tue, 22 Feb 2022 04:53:03 + UNCLASSIFIED Thank you for sharing your experience. I was thinking that DoM is built in feature and it can be enabled/disabled online for a certain directories. What do you mean by reformat to converting to DoM (or away from it). I think just Metadata target size is important. I also thought creating flash OST on metadata server. But I was not sure what to install on metadata server for this purpose. Can Metadata server be an OSS server at the same time? If it is possible I would prefer flash OST on Metadata server instead of DoM. Because Our metadata target size is small, it seems I have to do risky operations to expand size. imho, because of the less RPC traffic DoM shows more performance than flash OST. Am I right? Best Regards; From: Marion Hakanson mailto:hakan...@ohsu.edu><mailto:hakan...@ohsu.edu>> Sent: Thursday, February 17, 2022 8:20 PM To: Taner KARAGÖL mailto:kara...@aselsan.com.tr><mailto:kara...@aselsan.com.tr>> Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org> Subject: Re: [lustre-discuss] How to speed up Lustre We started with DoM on our new Lustre system a couple years ago. - Converting to DoM (or away from it) is a full-reformat operation. - DoM uses a fixed amount of metadata space (64k minimum for us) for every file, even those smaller than 64k. Basically, DoM uses a lot of flash metadata space, more than we planned for, and more than we could afford. We ended up switching to a PFL arrangement, where the first 64k lives on flash OST's (mounted on our metadata servers), and the remainder of larger files lives on HDD OST's. This is working very well for our small-file workloads, and uses less flash space than the DoM configuration did. Since you don't already have DoM in effect, it may be possible that you could add flash OST's, configure a PFL, and then use "lfs migrate" to re-layout existing files into the new OST's. Your mileage may vary, so be safe! Regards, Marion On Feb 14, 2022, at 03:32, Taner KARAGÖL via lustre-discuss mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org>> wrote:  UNCLASSIFIED Hi Everybody; We have a performance problem with small files on our HPC system (120 compute nodes). Our all OSS targets are classic spinning HDDs. To speed up, I want to configure Data on Metadata. Our metadata target has SDD disks. Underlying file systems are ZFS (for OSS and Meta) Lustre version: 2.12.5 ZFS version: .0.7.13 Our Lustre file system size is 720TB (2 OSS servers, 1 enclosure with 6 zpools), Metadata file system size is 2.1TB(1 enclosure and 1 metadata target). What is your opinions to speed up this setup? I want to configure DoM but I am concerning about Metadata size. My questions: 1. How can I increase Medatadata size? Metada
Re: [lustre-discuss] How to speed up Lustre
ating flash OST on metadata server. But I was not sure > > what > > to install on metadata server for this purpose. Can Metadata server be an > > OSS > > server at the same time? If it is possible I would prefer flash OST on > > Metadata server instead of DoM. Because Our metadata target size is small, > > it > > seems I have to do risky operations to expand size. > > Yes, our metadata servers are also OSS's at the same time. The flash > > OST's are separate volumes (and drives) from the MDT's, so less scary (:-). > > kara...@aselsan.com.tr<mailto:kara...@aselsan.com.tr> said: > > imho, because of the less RPC traffic DoM shows more performance than flash > > OST. Am I right? > > The documentation does say there that using DoM for small files will produce > > less RPC traffic than using OST's for small files. > > But as I said earlier, for us, the amount of flash needed to support DoM > > was a lot higher than with the flash OST approach (we have a high > > percentage, > > by number, of small files). > > I'll also note that we had a wish to mostly "set and forget" the layout > > for our Lustre filesystem. We have not figured out a way to predict > > or control where small files (or large ones) are going to end up, so > > trying to craft optimal layouts in particular directories for particular > > file sizes has turned out to not be feasible for us. PFL has been a > > win for us here, for that reason. > > Our conclusion was that in order to take advantage of the performance > > improvements of DoM, you need enough money for lots of flash, or you need > > enough staff time to manage the DoM layouts to fit into that flash. > > We have neither of those conditions, and we find that using PFL and > > flash OST's for small files is working very well for us. > > Regards, > > Marion > > From: =?utf-8?B?VGFuZXIgS0FSQUfDlkw=?= > > mailto:kara...@aselsan.com.tr>> > > To: Marion Hakanson mailto:hakan...@ohsu.edu>> > > CC: > > "lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>" > > mailto:lustre-discuss@lists.lustre.org>> > > Date: Tue, 22 Feb 2022 04:53:03 + > > > > UNCLASSIFIED > > > > Thank you for sharing your experience. > > > > I was thinking that DoM is built in feature and it can be enabled/disabled > > online for a certain directories. What do you mean by reformat to > > converting to DoM (or away from it). I think just Metadata target size is > > important. > > > > I also thought creating flash OST on metadata server. But I was not sure > > what to install on metadata server for this purpose. Can Metadata server be > > an OSS server at the same time? If it is possible I would prefer flash OST > > on Metadata server instead of DoM. Because Our metadata target size is > > small, it seems I have to do risky operations to expand size. > > > > imho, because of the less RPC traffic DoM shows more performance than flash > > OST. Am I right? > > > > Best Regards; > > > > > > From: Marion Hakanson mailto:hakan...@ohsu.edu>> > > Sent: Thursday, February 17, 2022 8:20 PM > > To: Taner KARAGÖL mailto:kara...@aselsan.com.tr>> > > Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> > > Subject: Re: [lustre-discuss] How to speed up Lustre > > > > We started with DoM on our new Lustre system a couple years ago. > >- Converting to DoM (or away from it) is a full-reformat operation. > >- DoM uses a fixed amount of metadata space (64k minimum for us) for > > every file, even those smaller than 64k. > > > > Basically, DoM uses a lot of flash metadata space, more than we planned > > for, and more than we could afford. > > > > We ended up switching to a PFL arrangement, where the first 64k lives on > > flash OST's (mounted on our metadata servers), and the remainder of larger > > files lives on HDD OST's. This is working very well for our small-file > > workloads, and uses less flash space than the DoM configuration did. > > > > Since you don't already have DoM in effect, it may be possible that you > > could add flash OST's, configure a PFL, and then use "lfs migrate" to > > re-layout existing files into the new OST's. Your mileage may vary, so be > > safe! > > > > Regards, > > > > Marion > > > > > > > > On Feb 14, 2022, at 03:32, Taner KARAGÖL via lustre-discu
Re: [lustre-discuss] How to speed up Lustre
ey for lots of flash, or you need enough staff time to manage the DoM layouts to fit into that flash. We have neither of those conditions, and we find that using PFL and flash OST's for small files is working very well for us. Regards, Marion From: =?utf-8?B?VGFuZXIgS0FSQUfDlkw=?= mailto:kara...@aselsan.com.tr>> To: Marion Hakanson mailto:hakan...@ohsu.edu>> CC: "lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>" mailto:lustre-discuss@lists.lustre.org>> Date: Tue, 22 Feb 2022 04:53:03 + UNCLASSIFIED Thank you for sharing your experience. I was thinking that DoM is built in feature and it can be enabled/disabled online for a certain directories. What do you mean by reformat to converting to DoM (or away from it). I think just Metadata target size is important. I also thought creating flash OST on metadata server. But I was not sure what to install on metadata server for this purpose. Can Metadata server be an OSS server at the same time? If it is possible I would prefer flash OST on Metadata server instead of DoM. Because Our metadata target size is small, it seems I have to do risky operations to expand size. imho, because of the less RPC traffic DoM shows more performance than flash OST. Am I right? Best Regards; From: Marion Hakanson mailto:hakan...@ohsu.edu>> Sent: Thursday, February 17, 2022 8:20 PM To: Taner KARAGÖL mailto:kara...@aselsan.com.tr>> Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> Subject: Re: [lustre-discuss] How to speed up Lustre We started with DoM on our new Lustre system a couple years ago. - Converting to DoM (or away from it) is a full-reformat operation. - DoM uses a fixed amount of metadata space (64k minimum for us) for every file, even those smaller than 64k. Basically, DoM uses a lot of flash metadata space, more than we planned for, and more than we could afford. We ended up switching to a PFL arrangement, where the first 64k lives on flash OST's (mounted on our metadata servers), and the remainder of larger files lives on HDD OST's. This is working very well for our small-file workloads, and uses less flash space than the DoM configuration did. Since you don't already have DoM in effect, it may be possible that you could add flash OST's, configure a PFL, and then use "lfs migrate" to re-layout existing files into the new OST's. Your mileage may vary, so be safe! Regards, Marion On Feb 14, 2022, at 03:32, Taner KARAGÖL via lustre-discuss mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org>> wrote:  UNCLASSIFIED Hi Everybody; We have a performance problem with small files on our HPC system (120 compute nodes). Our all OSS targets are classic spinning HDDs. To speed up, I want to configure Data on Metadata. Our metadata target has SDD disks. Underlying file systems are ZFS (for OSS and Meta) Lustre version: 2.12.5 ZFS version: .0.7.13 Our Lustre file system size is 720TB (2 OSS servers, 1 enclosure with 6 zpools), Metadata file system size is 2.1TB(1 enclosure and 1 metadata target). What is your opinions to speed up this setup? I want to configure DoM but I am concerning about Metadata size. My questions: 1. How can I increase Medatadata size? Metadata enclosure has a empty slots. Is there a way to increase size online/offline? 2. Is it possible to migrate big files from DoM to OSS targets completely? Off course online migration. (So I think I can free Metadata for new small files). Best Regards; Taner Dikkat: Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir. Attention: This e-mail message is privileged and confidential. If you are not the intended recipient please delete the message and notify the sender. E-mails to and from the company are monitored for operational reasons and in accordance with lawful business practices. Any views or opinions presented are solely those of the author and do not necessarily represent the views of the company. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org> https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mi0JBg!bW2FnSTRNdX7DpkjIiMayeexmYJ3D5Xt7wtneny2zgGi1ZXPcy7QMRlM3mno-HWR$<https://urldefense.com/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mi0JBg!bW2FnSTRNdX7
Re: [lustre-discuss] How to speed up Lustre
t in feature and it can be enabled/disabled online for a certain directories. What do you mean by reformat to converting to DoM (or away from it). I think just Metadata target size is important. I also thought creating flash OST on metadata server. But I was not sure what to install on metadata server for this purpose. Can Metadata server be an OSS server at the same time? If it is possible I would prefer flash OST on Metadata server instead of DoM. Because Our metadata target size is small, it seems I have to do risky operations to expand size. imho, because of the less RPC traffic DoM shows more performance than flash OST. Am I right? Best Regards; From: Marion Hakanson mailto:hakan...@ohsu.edu>> Sent: Thursday, February 17, 2022 8:20 PM To: Taner KARAGÖL mailto:kara...@aselsan.com.tr>> Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> Subject: Re: [lustre-discuss] How to speed up Lustre We started with DoM on our new Lustre system a couple years ago. - Converting to DoM (or away from it) is a full-reformat operation. - DoM uses a fixed amount of metadata space (64k minimum for us) for every file, even those smaller than 64k. Basically, DoM uses a lot of flash metadata space, more than we planned for, and more than we could afford. We ended up switching to a PFL arrangement, where the first 64k lives on flash OST's (mounted on our metadata servers), and the remainder of larger files lives on HDD OST's. This is working very well for our small-file workloads, and uses less flash space than the DoM configuration did. Since you don't already have DoM in effect, it may be possible that you could add flash OST's, configure a PFL, and then use "lfs migrate" to re-layout existing files into the new OST's. Your mileage may vary, so be safe! Regards, Marion On Feb 14, 2022, at 03:32, Taner KARAGÖL via lustre-discuss mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org>> wrote:  UNCLASSIFIED Hi Everybody; We have a performance problem with small files on our HPC system (120 compute nodes). Our all OSS targets are classic spinning HDDs. To speed up, I want to configure Data on Metadata. Our metadata target has SDD disks. Underlying file systems are ZFS (for OSS and Meta) Lustre version: 2.12.5 ZFS version: .0.7.13 Our Lustre file system size is 720TB (2 OSS servers, 1 enclosure with 6 zpools), Metadata file system size is 2.1TB(1 enclosure and 1 metadata target). What is your opinions to speed up this setup? I want to configure DoM but I am concerning about Metadata size. My questions: 1. How can I increase Medatadata size? Metadata enclosure has a empty slots. Is there a way to increase size online/offline? 2. Is it possible to migrate big files from DoM to OSS targets completely? Off course online migration. (So I think I can free Metadata for new small files). Best Regards; Taner Dikkat: Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir. Attention: This e-mail message is privileged and confidential. If you are not the intended recipient please delete the message and notify the sender. E-mails to and from the company are monitored for operational reasons and in accordance with lawful business practices. Any views or opinions presented are solely those of the author and do not necessarily represent the views of the company. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org> https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mi0JBg!bW2FnSTRNdX7DpkjIiMayeexmYJ3D5Xt7wtneny2zgGi1ZXPcy7QMRlM3mno-HWR$<https://urldefense.com/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mi0JBg!bW2FnSTRNdX7DpkjIiMayeexmYJ3D5Xt7wtneny2zgGi1ZXPcy7QMRlM3mno-HWR$> ## Dikkat: Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir. ## Attention: This e-mail message is privileged and confidential. If you are not the intended recipi
Re: [lustre-discuss] How to speed up Lustre
Hi Marion, I do not fully understand how to "mount flash OSTs on a metadata server" - You have a couple of SSDs, you assemble these into on block device and format it with "mkfs.lustre --ost ..." ? And then mount it just as any other OST? - PFL then puts the first 64k on these OSTs and the rest of all files on the HDD-based OSTs? So, no magic on the MDS? I'm asking because we are considering something similar, but we would not have these flash-OSTs in the MDS-hardware but on separate OSS servers. Regards, Thomas On 23/02/2022 04.35, Marion Hakanson via lustre-discuss wrote: Hi again, kara...@aselsan.com.tr said: I was thinking that DoM is built in feature and it can be enabled/disabled online for a certain directories. What do you mean by reformat to converting to DoM (or away from it). I think just Metadata target size is important. When we first turned on DoM, it's likely that our Lustre system was old enough to need to be reformatted in order to support it. Our flash storage RAID configuration also needed to be expanded, but the system was not yet in production so a reformat was no big deal at the time. So perhaps your system will not be subject to this requirement (other than expanding your MDT flash somehow). kara...@aselsan.com.tr said: I also thought creating flash OST on metadata server. But I was not sure what to install on metadata server for this purpose. Can Metadata server be an OSS server at the same time? If it is possible I would prefer flash OST on Metadata server instead of DoM. Because Our metadata target size is small, it seems I have to do risky operations to expand size. Yes, our metadata servers are also OSS's at the same time. The flash OST's are separate volumes (and drives) from the MDT's, so less scary (:-). kara...@aselsan.com.tr said: imho, because of the less RPC traffic DoM shows more performance than flash OST. Am I right? The documentation does say there that using DoM for small files will produce less RPC traffic than using OST's for small files. But as I said earlier, for us, the amount of flash needed to support DoM was a lot higher than with the flash OST approach (we have a high percentage, by number, of small files). I'll also note that we had a wish to mostly "set and forget" the layout for our Lustre filesystem. We have not figured out a way to predict or control where small files (or large ones) are going to end up, so trying to craft optimal layouts in particular directories for particular file sizes has turned out to not be feasible for us. PFL has been a win for us here, for that reason. Our conclusion was that in order to take advantage of the performance improvements of DoM, you need enough money for lots of flash, or you need enough staff time to manage the DoM layouts to fit into that flash. We have neither of those conditions, and we find that using PFL and flash OST's for small files is working very well for us. Regards, Marion From: =?utf-8?B?VGFuZXIgS0FSQUfDlkw=?= To: Marion Hakanson CC: "lustre-discuss@lists.lustre.org" Date: Tue, 22 Feb 2022 04:53:03 + UNCLASSIFIED Thank you for sharing your experience. I was thinking that DoM is built in feature and it can be enabled/disabled online for a certain directories. What do you mean by reformat to converting to DoM (or away from it). I think just Metadata target size is important. I also thought creating flash OST on metadata server. But I was not sure what to install on metadata server for this purpose. Can Metadata server be an OSS server at the same time? If it is possible I would prefer flash OST on Metadata server instead of DoM. Because Our metadata target size is small, it seems I have to do risky operations to expand size. imho, because of the less RPC traffic DoM shows more performance than flash OST. Am I right? Best Regards; From: Marion Hakanson Sent: Thursday, February 17, 2022 8:20 PM To: Taner KARAGÖL Cc: lustre-discuss@lists.lustre.org Subject: Re: [lustre-discuss] How to speed up Lustre We started with DoM on our new Lustre system a couple years ago. - Converting to DoM (or away from it) is a full-reformat operation. - DoM uses a fixed amount of metadata space (64k minimum for us) for every file, even those smaller than 64k. Basically, DoM uses a lot of flash metadata space, more than we planned for, and more than we could afford. We ended up switching to a PFL arrangement, where the first 64k lives on flash OST's (mounted on our metadata servers), and the remainder of larger files lives on HDD OST's. This is working very well for our small-file workloads, and uses less flash space than the DoM configuration did. Since you don't already have DoM in effect, it may be possible that you could add flash OST's, configure a PFL, and then use "lfs migrate" to re-layout existing files into the new OST&#
Re: [lustre-discuss] How to speed up Lustre
Hi again, kara...@aselsan.com.tr said: > I was thinking that DoM is built in feature and it can be enabled/disabled > online for a certain directories. What do you mean by reformat to converting > to DoM (or away from it). I think just Metadata target size is important. When we first turned on DoM, it's likely that our Lustre system was old enough to need to be reformatted in order to support it. Our flash storage RAID configuration also needed to be expanded, but the system was not yet in production so a reformat was no big deal at the time. So perhaps your system will not be subject to this requirement (other than expanding your MDT flash somehow). kara...@aselsan.com.tr said: > I also thought creating flash OST on metadata server. But I was not sure what > to install on metadata server for this purpose. Can Metadata server be an OSS > server at the same time? If it is possible I would prefer flash OST on > Metadata server instead of DoM. Because Our metadata target size is small, it > seems I have to do risky operations to expand size. Yes, our metadata servers are also OSS's at the same time. The flash OST's are separate volumes (and drives) from the MDT's, so less scary (:-). kara...@aselsan.com.tr said: > imho, because of the less RPC traffic DoM shows more performance than flash > OST. Am I right? The documentation does say there that using DoM for small files will produce less RPC traffic than using OST's for small files. But as I said earlier, for us, the amount of flash needed to support DoM was a lot higher than with the flash OST approach (we have a high percentage, by number, of small files). I'll also note that we had a wish to mostly "set and forget" the layout for our Lustre filesystem. We have not figured out a way to predict or control where small files (or large ones) are going to end up, so trying to craft optimal layouts in particular directories for particular file sizes has turned out to not be feasible for us. PFL has been a win for us here, for that reason. Our conclusion was that in order to take advantage of the performance improvements of DoM, you need enough money for lots of flash, or you need enough staff time to manage the DoM layouts to fit into that flash. We have neither of those conditions, and we find that using PFL and flash OST's for small files is working very well for us. Regards, Marion > From: =?utf-8?B?VGFuZXIgS0FSQUfDlkw=?= > To: Marion Hakanson > CC: "lustre-discuss@lists.lustre.org" > Date: Tue, 22 Feb 2022 04:53:03 + > > UNCLASSIFIED > > Thank you for sharing your experience. > > I was thinking that DoM is built in feature and it can be enabled/disabled > online for a certain directories. What do you mean by reformat to converting > to DoM (or away from it). I think just Metadata target size is important. > > I also thought creating flash OST on metadata server. But I was not sure what > to install on metadata server for this purpose. Can Metadata server be an OSS > server at the same time? If it is possible I would prefer flash OST on > Metadata server instead of DoM. Because Our metadata target size is small, it > seems I have to do risky operations to expand size. > > imho, because of the less RPC traffic DoM shows more performance than flash > OST. Am I right? > > Best Regards; > > > From: Marion Hakanson > Sent: Thursday, February 17, 2022 8:20 PM > To: Taner KARAGÖL > Cc: lustre-discuss@lists.lustre.org > Subject: Re: [lustre-discuss] How to speed up Lustre > > We started with DoM on our new Lustre system a couple years ago. > - Converting to DoM (or away from it) is a full-reformat operation. > - DoM uses a fixed amount of metadata space (64k minimum for us) for every > file, even those smaller than 64k. > > Basically, DoM uses a lot of flash metadata space, more than we planned for, > and more than we could afford. > > We ended up switching to a PFL arrangement, where the first 64k lives on > flash OST's (mounted on our metadata servers), and the remainder of larger > files lives on HDD OST's. This is working very well for our small-file > workloads, and uses less flash space than the DoM configuration did. > > Since you don't already have DoM in effect, it may be possible that you could > add flash OST's, configure a PFL, and then use "lfs migrate" to re-layout > existing files into the new OST's. Your mileage may vary, so be safe! > > Regards, > > Marion > > > > On Feb 14, 2022, at 03:32, Taner KARAGÖL via lustre-discuss > mailto:lustre-discuss@lists.lustre.org>> > wrote: >  > UNCLASSIFIED > > Hi Everybody; > > We have a performance problem with small files on
Re: [lustre-discuss] How to speed up Lustre
UNCLASSIFIED Thank you for sharing your experience. I was thinking that DoM is built in feature and it can be enabled/disabled online for a certain directories. What do you mean by reformat to converting to DoM (or away from it). I think just Metadata target size is important. I also thought creating flash OST on metadata server. But I was not sure what to install on metadata server for this purpose. Can Metadata server be an OSS server at the same time? If it is possible I would prefer flash OST on Metadata server instead of DoM. Because Our metadata target size is small, it seems I have to do risky operations to expand size. imho, because of the less RPC traffic DoM shows more performance than flash OST. Am I right? Best Regards; From: Marion Hakanson Sent: Thursday, February 17, 2022 8:20 PM To: Taner KARAGÖL Cc: lustre-discuss@lists.lustre.org Subject: Re: [lustre-discuss] How to speed up Lustre We started with DoM on our new Lustre system a couple years ago. - Converting to DoM (or away from it) is a full-reformat operation. - DoM uses a fixed amount of metadata space (64k minimum for us) for every file, even those smaller than 64k. Basically, DoM uses a lot of flash metadata space, more than we planned for, and more than we could afford. We ended up switching to a PFL arrangement, where the first 64k lives on flash OST's (mounted on our metadata servers), and the remainder of larger files lives on HDD OST's. This is working very well for our small-file workloads, and uses less flash space than the DoM configuration did. Since you don't already have DoM in effect, it may be possible that you could add flash OST's, configure a PFL, and then use "lfs migrate" to re-layout existing files into the new OST's. Your mileage may vary, so be safe! Regards, Marion On Feb 14, 2022, at 03:32, Taner KARAGÖL via lustre-discuss mailto:lustre-discuss@lists.lustre.org>> wrote: UNCLASSIFIED Hi Everybody; We have a performance problem with small files on our HPC system (120 compute nodes). Our all OSS targets are classic spinning HDDs. To speed up, I want to configure Data on Metadata. Our metadata target has SDD disks. Underlying file systems are ZFS (for OSS and Meta) Lustre version: 2.12.5 ZFS version: .0.7.13 Our Lustre file system size is 720TB (2 OSS servers, 1 enclosure with 6 zpools), Metadata file system size is 2.1TB(1 enclosure and 1 metadata target). What is your opinions to speed up this setup? I want to configure DoM but I am concerning about Metadata size. My questions: 1. How can I increase Medatadata size? Metadata enclosure has a empty slots. Is there a way to increase size online/offline? 2. Is it possible to migrate big files from DoM to OSS targets completely? Off course online migration. (So I think I can free Metadata for new small files). Best Regards; Taner Dikkat: Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir. Attention: This e-mail message is privileged and confidential. If you are not the intended recipient please delete the message and notify the sender. E-mails to and from the company are monitored for operational reasons and in accordance with lawful business practices. Any views or opinions presented are solely those of the author and do not necessarily represent the views of the company. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mi0JBg!bW2FnSTRNdX7DpkjIiMayeexmYJ3D5Xt7wtneny2zgGi1ZXPcy7QMRlM3mno-HWR$<https://urldefense.com/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mi0JBg!bW2FnSTRNdX7DpkjIiMayeexmYJ3D5Xt7wtneny2zgGi1ZXPcy7QMRlM3mno-HWR$> ## Dikkat: Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir. ## Attention: This e-mail message is privileged and confidential. If you are not the intended recipient please delete the message and notify the sender. E-mails to and from the company are monitored for opera
Re: [lustre-discuss] How to speed up Lustre
We started with DoM on our new Lustre system a couple years ago. - Converting to DoM (or away from it) is a full-reformat operation. - DoM uses a fixed amount of metadata space (64k minimum for us) for every file, even those smaller than 64k. Basically, DoM uses a lot of flash metadata space, more than we planned for, and more than we could afford. We ended up switching to a PFL arrangement, where the first 64k lives on flash OST's (mounted on our metadata servers), and the remainder of larger files lives on HDD OST's. This is working very well for our small-file workloads, and uses less flash space than the DoM configuration did. Since you don't already have DoM in effect, it may be possible that you could add flash OST's, configure a PFL, and then use "lfs migrate" to re-layout existing files into the new OST's. Your mileage may vary, so be safe! Regards, Marion On Feb 14, 2022, at 03:32, Taner KARAGÖL via lustre-discuss wrote: UNCLASSIFIED Hi Everybody; We have a performance problem with small files on our HPC system (120 compute nodes). Our all OSS targets are classic spinning HDDs. To speed up, I want to configure Data on Metadata. Our metadata target has SDD disks. Underlying file systems are ZFS (for OSS and Meta) Lustre version: 2.12.5 ZFS version: .0.7.13 Our Lustre file system size is 720TB (2 OSS servers, 1 enclosure with 6 zpools), Metadata file system size is 2.1TB(1 enclosure and 1 metadata target). What is your opinions to speed up this setup? I want to configure DoM but I am concerning about Metadata size. My questions: 1. How can I increase Medatadata size? Metadata enclosure has a empty slots. Is there a way to increase size online/offline? 2. Is it possible to migrate big files from DoM to OSS targets completely? Off course online migration. (So I think I can free Metadata for new small files). Best Regards; Taner Dikkat: Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir. Attention: This e-mail message is privileged and confidential. If you are not the intended recipient please delete the message and notify the sender. E-mails to and from the company are monitored for operational reasons and in accordance with lawful business practices. Any views or opinions presented are solely those of the author and do not necessarily represent the views of the company. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mi0JBg!bW2FnSTRNdX7DpkjIiMayeexmYJ3D5Xt7wtneny2zgGi1ZXPcy7QMRlM3mno-HWR$ ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] How to speed up Lustre
UNCLASSIFIED Hi Everybody; We have a performance problem with small files on our HPC system (120 compute nodes). Our all OSS targets are classic spinning HDDs. To speed up, I want to configure Data on Metadata. Our metadata target has SDD disks. Underlying file systems are ZFS (for OSS and Meta) Lustre version: 2.12.5 ZFS version: .0.7.13 Our Lustre file system size is 720TB (2 OSS servers, 1 enclosure with 6 zpools), Metadata file system size is 2.1TB(1 enclosure and 1 metadata target). What is your opinions to speed up this setup? I want to configure DoM but I am concerning about Metadata size. My questions: 1. How can I increase Medatadata size? Metadata enclosure has a empty slots. Is there a way to increase size online/offline? 2. Is it possible to migrate big files from DoM to OSS targets completely? Off course online migration. (So I think I can free Metadata for new small files). Best Regards; Taner ## Dikkat: Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir. ## Attention: This e-mail message is privileged and confidential. If you are not the intended recipient please delete the message and notify the sender. E-mails to and from the company are monitored for operational reasons and in accordance with lawful business practices. Any views or opinions presented are solely those of the author and do not necessarily represent the views of the company. ## ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
