Re: [lustre-discuss] Extending Lustre file system

[Andreas Dilger via lustre-discuss]

I would recommend *against* mounting all 175 OSTs at the same time.  There are 
(or at least were*) some issues with the MGS registration RPCs timing out when 
too many config changes happen at once.  Your "mount and wait 2 sec" is more 
robust and doesn't take very much time (a few minutes) vs. having to restart if 
some of the OSTs have problems registering.  Also, the config logs will have 
the OSTs in a nice order, which doesn't affect any functionality, but makes it 
easier for the admin to see if some device is connected in "lctl dl" output.

Cheers, Andreas


[*] some fixes have landed over time to improve registration RPC resend.

On Jan 8, 2024, at 11:57, Thomas Roth via lustre-discuss 
mailto:lustre-discuss@lists.lustre.org>> wrote:

Yes, sorry, I meant the actual procedure of mounting the OSTs for the first 
time.

Last year I did that with 175 OSTs - replacements for EOL hardware. All OSTs 
had been formatted with a specific index, so probably creating a suitable 
/etc/fstab everywhere and sending a 'mount -a -t lustre' to all OSTs 
simultaneously would have worked.

But why the hurry? Instead, I logged in to my new OSS, mounted the OSTs with 2 
sec between each mount command, watched the OSS log, watched the MDS log, saw 
the expected log messages, proceeded to the new OSS - all fine ;-)  Such a 
leisurely approach takes its time, of course.

Once all OSTs were happily incorporated, we put the max_create_count (set to 0 
before) to some finite value and started file migration. As long as the 
migration is more effective, faster, than the users's file creations, the 
result should be evenly filled OSTs with a good mixture of files (file sizes, 
ages, types).


Cheers
Thomas

On 1/8/24 19:07, Andreas Dilger wrote:
The need to rebalance depends on how full the existing OSTs are.  My 
recommendation if you know that the data will continue to grow is to add new 
OSTs when the existing ones are at 60-70% full, and add them in larger groups 
rather than one at a time.
Cheers, Andreas
On Jan 8, 2024, at 09:29, Thomas Roth via lustre-discuss 
mailto:lustre-discuss@lists.lustre.org>> wrote:

Just mount the OSTs, one by one and perhaps not if your system is heavily 
loaded. Follow what happens in the MDS log and the OSS log.
And try to rebalance the OSTs fill levels afterwards - very empty OSTs will 
attract all new files, which might be hot and direct your users's fire to your 
new OSS only.

Regards,
Thomas

On 1/8/24 15:38, Backer via lustre-discuss wrote:
Hi,
Good morning and happy new year!
I have a quick question on extending a lustre file system. The extension is 
performed online. I am looking for any best practices or anything to watchout 
while doing the file system extension. The file system extension is done adding 
new OSS and many OSTs within these servers.
Really appreciate your help on this.
Regards,
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud







___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Extending Lustre file system

[Thomas Roth via lustre-discuss]

Yes, sorry, I meant the actual procedure of mounting the OSTs for the first 
time.

Last year I did that with 175 OSTs - replacements for EOL hardware. All OSTs had been formatted with a specific index, so probably creating a suitable 
/etc/fstab everywhere and sending a 'mount -a -t lustre' to all OSTs simultaneously would have worked.


But why the hurry? Instead, I logged in to my new OSS, mounted the OSTs with 2 sec between each mount command, watched the OSS log, watched the MDS 
log, saw the expected log messages, proceeded to the new OSS - all fine ;-)  Such a leisurely approach takes its time, of course.


Once all OSTs were happily incorporated, we put the max_create_count (set to 0 before) to some finite value and started file migration. As long as the 
migration is more effective, faster, than the users's file creations, the result should be evenly filled OSTs with a good mixture of files (file 
sizes, ages, types).



Cheers
Thomas

On 1/8/24 19:07, Andreas Dilger wrote:

The need to rebalance depends on how full the existing OSTs are.  My 
recommendation if you know that the data will continue to grow is to add new 
OSTs when the existing ones are at 60-70% full, and add them in larger groups 
rather than one at a time.

Cheers, Andreas


On Jan 8, 2024, at 09:29, Thomas Roth via lustre-discuss 
 wrote:

Just mount the OSTs, one by one and perhaps not if your system is heavily 
loaded. Follow what happens in the MDS log and the OSS log.
And try to rebalance the OSTs fill levels afterwards - very empty OSTs will 
attract all new files, which might be hot and direct your users's fire to your 
new OSS only.

Regards,
Thomas


On 1/8/24 15:38, Backer via lustre-discuss wrote:
Hi,
Good morning and happy new year!
I have a quick question on extending a lustre file system. The extension is 
performed online. I am looking for any best practices or anything to watchout 
while doing the file system extension. The file system extension is done adding 
new OSS and many OSTs within these servers.
Really appreciate your help on this.
Regards,
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] [EXTERNAL] [BULK] MDS hardware - NVME?

[Jeff Johnson]

Today nvme/mdraid/ldiskfs will beat nvme/zfs on MDS IOPs but you can
close the gap somewhat with tuning, zfs ashift/recordsize and special
allocation class vdevs. While the IOPs performance favors
nvme/mdraid/ldiskfs there are tradeoffs. The snapshot/backup abilities
of ZFS and the security it provides to the most critical function in a
Lustre file system shouldn't be undervalued. From personal experience,
I'd much rather deal with zfs in the event of a seriously jackknifed
MDT than mdraid/ldiskfs and both zfs and mdraid/ldiskfs are preferable
to trying to unscramble a vendor blackbox hwraid volume. ;-)

When zfs directio lands and is fully integrated into Lustre the
performance differences *should* be negligible.

Just my $.02 worth

On Mon, Jan 8, 2024 at 8:23 AM Thomas Roth via lustre-discuss
 wrote:
>
> Hi Cameron,
>
> did you run a performance comparison between ZFS and mdadm-raid on the MDTs?
> I'm currently doing some tests, and the results favor software raid, in 
> particular when it comes to IOPS.
>
> Regards
> Thomas
>
> On 1/5/24 19:55, Cameron Harr via lustre-discuss wrote:
> > This doesn't answer your question about ldiskfs on zvols, but we've been 
> > running MDTs on ZFS on NVMe in production for a couple years (and on SAS 
> > SSDs for many years prior). Our current production MDTs using NVMe consist 
> > of one zpool/node made up of 3x 2-drive mirrors, but we've been 
> > experimenting lately with using raidz3 and possibly even raidz2 for MDTs 
> > since SSDs have been pretty reliable for us.
> >
> > Cameron
> >
> > On 1/5/24 9:07 AM, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] 
> > via lustre-discuss wrote:
> >> We are in the process of retiring two long standing LFS's (about 8 years 
> >> old), which we built and managed ourselves.  Both use ZFS and have the 
> >> MDT'S on ssd's in a JBOD that require the kind of software-based 
> >> management you describe, in our case ZFS pools built on multipath devices. 
> >>  The MDT in one is ZFS and the MDT in the other LFS is ldiskfs but uses 
> >> ZFS and a zvol as you describe - we build the ldiskfs MDT on top of the 
> >> zvol.  Generally, this has worked well for us, with one big caveat.  If 
> >> you look for my posts to this list and the ZFS list you'll find more 
> >> details.  The short version is that we utilize ZFS snapshots and clones to 
> >> do backups of the metadata.  We've run into situations where the backup 
> >> process stalls, leaving a clone hanging around.  We've experienced a 
> >> situation a couple of times where the clone and the primary zvol get 
> >> swapped, effectively rolling back our metadata to the point when the clone 
> >> was created.  I have tried, unsuccessfully, to recreate
> >> that in a test environment.  So if you do that kind of setup, make sure 
> >> you have good monitoring in place to detect if your backups/clones stall.  
> >> We've kept up with lustre and ZFS updates over the years and are currently 
> >> on lustre 2.14 and ZFS 2.1.  We've seen the gap between our ZFS MDT and 
> >> ldiskfs performance shrink to the point where they are pretty much on par 
> >> to each now.  I think our ZFS MDT performance could be better with more 
> >> hardware and software tuning but our small team hasn't had the bandwidth 
> >> to tackle that.
> >>
> >> Our newest LFS is vendor provided and uses NVMe MDT's.  I'm not at liberty 
> >> to talk about the proprietary way those devices are managed.  However, the 
> >> metadata performance is SO much better than our older LFS's, for a lot of 
> >> reasons, but I'd highly recommend NVMe's for your MDT's.
> >>
> >> -Original Message-
> >> From: lustre-discuss  >> > on behalf of Thomas Roth 
> >> via lustre-discuss  >> >
> >> Reply-To: Thomas Roth mailto:t.r...@gsi.de>>
> >> Date: Friday, January 5, 2024 at 9:03 AM
> >> To: Lustre Diskussionsliste  >> >
> >> Subject: [EXTERNAL] [BULK] [lustre-discuss] MDS hardware - NVME?
> >>
> >>
> >> CAUTION: This email originated from outside of NASA. Please take care when 
> >> clicking links or opening attachments. Use the "Report Message" button to 
> >> report suspicious messages to the NASA SOC.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Dear all,
> >>
> >>
> >> considering NVME storage for the next MDS.
> >>
> >>
> >> As I understand, NVME disks are bundled in software, not by a hardware 
> >> raid controller.
> >> This would be done using Linux software raid, mdadm, correct?
> >>
> >>
> >> We have some experience with ZFS, which we use on our OSTs.
> >> But I would like to stick to ldiskfs for the MDTs, and a zpool with a zvol 
> >> on top which is then formatted with ldiskfs - to much voodoo...
> >>
> >>
> >> How is this handled elsewhere? Any experiences?
> >>
> >>
> >>
> >>
> >> The available devices are quite large. If I create a raid-10 out of 4 
> >> disks, e.g. 7 TB each, 

Re: [lustre-discuss] Extending Lustre file system

[Andreas Dilger via lustre-discuss]

The need to rebalance depends on how full the existing OSTs are.  My 
recommendation if you know that the data will continue to grow is to add new 
OSTs when the existing ones are at 60-70% full, and add them in larger groups 
rather than one at a time.

Cheers, Andreas

> On Jan 8, 2024, at 09:29, Thomas Roth via lustre-discuss 
>  wrote:
> 
> Just mount the OSTs, one by one and perhaps not if your system is heavily 
> loaded. Follow what happens in the MDS log and the OSS log.
> And try to rebalance the OSTs fill levels afterwards - very empty OSTs will 
> attract all new files, which might be hot and direct your users's fire to 
> your new OSS only.
> 
> Regards,
> Thomas
> 
>> On 1/8/24 15:38, Backer via lustre-discuss wrote:
>> Hi,
>> Good morning and happy new year!
>> I have a quick question on extending a lustre file system. The extension is 
>> performed online. I am looking for any best practices or anything to 
>> watchout while doing the file system extension. The file system extension is 
>> done adding new OSS and many OSTs within these servers.
>> Really appreciate your help on this.
>> Regards,
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Extending Lustre file system

[Thomas Roth via lustre-discuss]

Just mount the OSTs, one by one and perhaps not if your system is heavily 
loaded. Follow what happens in the MDS log and the OSS log.
And try to rebalance the OSTs fill levels afterwards - very empty OSTs will 
attract all new files, which might be hot and direct your users's fire to your 
new OSS only.

Regards,
Thomas

On 1/8/24 15:38, Backer via lustre-discuss wrote:

Hi,

Good morning and happy new year!

I have a quick question on extending a lustre file system. The extension is 
performed online. I am looking for any best practices or anything to watchout 
while doing the file system extension. The file system extension is done adding 
new OSS and many OSTs within these servers.

Really appreciate your help on this.

Regards,




___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] [EXTERNAL] [BULK] MDS hardware - NVME?

[Thomas Roth via lustre-discuss]

Hi Cameron,

did you run a performance comparison between ZFS and mdadm-raid on the MDTs?
I'm currently doing some tests, and the results favor software raid, in 
particular when it comes to IOPS.

Regards
Thomas

On 1/5/24 19:55, Cameron Harr via lustre-discuss wrote:

This doesn't answer your question about ldiskfs on zvols, but we've been 
running MDTs on ZFS on NVMe in production for a couple years (and on SAS SSDs 
for many years prior). Our current production MDTs using NVMe consist of one 
zpool/node made up of 3x 2-drive mirrors, but we've been experimenting lately 
with using raidz3 and possibly even raidz2 for MDTs since SSDs have been pretty 
reliable for us.

Cameron

On 1/5/24 9:07 AM, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via 
lustre-discuss wrote:
We are in the process of retiring two long standing LFS's (about 8 years old), which we built and managed ourselves.  Both use ZFS and have the MDT'S on ssd's in a JBOD that require the kind of software-based management you describe, in our case ZFS pools built on multipath devices.  The MDT in one is ZFS and the MDT in the other LFS is ldiskfs but uses ZFS and a zvol as you describe - we build the ldiskfs MDT on top of the zvol.  Generally, this has worked well for us, with one big caveat.  If you look for my posts to this list and the ZFS list you'll find more details.  The short version is that we utilize ZFS snapshots and clones to do backups of the metadata.  We've run into situations where the backup process stalls, leaving a clone hanging around.  We've experienced a situation a couple of times where the clone and the primary zvol get swapped, effectively rolling back our metadata to the point when the clone was created.  I have tried, unsuccessfully, to recreate 
that in a test environment.  So if you do that kind of setup, make sure you have good monitoring in place to detect if your backups/clones stall.  We've kept up with lustre and ZFS updates over the years and are currently on lustre 2.14 and ZFS 2.1.  We've seen the gap between our ZFS MDT and ldiskfs performance shrink to the point where they are pretty much on par to each now.  I think our ZFS MDT performance could be better with more hardware and software tuning but our small team hasn't had the bandwidth to tackle that.


Our newest LFS is vendor provided and uses NVMe MDT's.  I'm not at liberty to 
talk about the proprietary way those devices are managed.  However, the 
metadata performance is SO much better than our older LFS's, for a lot of 
reasons, but I'd highly recommend NVMe's for your MDT's.

-Original Message-
From: lustre-discuss mailto:lustre-discuss-boun...@lists.lustre.org>> on behalf of Thomas Roth via lustre-discuss 
mailto:lustre-discuss@lists.lustre.org>>
Reply-To: Thomas Roth mailto:t.r...@gsi.de>>
Date: Friday, January 5, 2024 at 9:03 AM
To: Lustre Diskussionsliste mailto:lustre-discuss@lists.lustre.org>>
Subject: [EXTERNAL] [BULK] [lustre-discuss] MDS hardware - NVME?


CAUTION: This email originated from outside of NASA. Please take care when clicking links 
or opening attachments. Use the "Report Message" button to report suspicious 
messages to the NASA SOC.








Dear all,


considering NVME storage for the next MDS.


As I understand, NVME disks are bundled in software, not by a hardware raid 
controller.
This would be done using Linux software raid, mdadm, correct?


We have some experience with ZFS, which we use on our OSTs.
But I would like to stick to ldiskfs for the MDTs, and a zpool with a zvol on 
top which is then formatted with ldiskfs - to much voodoo...


How is this handled elsewhere? Any experiences?




The available devices are quite large. If I create a raid-10 out of 4 disks, 
e.g. 7 TB each, my MDT will be 14 TB - already close to the 16 TB limit.
So no need for a box with lots of U.3 slots.


But for MDS operations, we will still need a powerful dual-CPU system with lots 
of RAM.
Then the NVME devices should be distributed between the CPUs?
Is there a way to pinpoint this in a call for tender?




Best regards,
Thomas



Thomas Roth


GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, 
https://urldefense.us/v3/__http://www.gsi.de/__;!!G2kpM7uM-TzIFchu!1QmOnUbmSPpZPcc39XFZ3S-Vk4Dmh-Q78Gpm8ylYUf6Zhv_zpb2VXkM4C5Uhh05x01MhjqJTYZ5boqzEhkx6JF_rGY74EQ$
  



Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
State Secretary / Staatssekretär Dr. Volkmar Dietz




___
lustre-discuss mailing list

Re: [lustre-discuss] [EXTERNAL] [BULK] MDS hardware - NVME?

[Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss]

Our setup has a single JBOD connected to 2 servers but the JBOD has dual 
controllers.  Each server connects to both controllers for redundancy so there 
are 4 connections to each server.  So we have a paired HA setup where one peer 
node can take over the OSTs/MDTs of its peer node.  Some specifics on our 
hardware:

Supermicro twin servers:
https://www.supermicro.com/products/archive/system/sys-6027tr-d71frf

JBOD:
https://www.supermicro.com/products/archive/chassis/sc946ed-r2kjbod

Each pair can “zpool import” all pools from either pair.  Here is an excerpt 
from our ldev.conf file


#local  foreign/-  label   [md|zfs:]device-path   [journal-path]/- [raidtab]

# primary hpfs-fsl (aka /nobackup) lustre file system
hpfs-fsl-mds0.fsl.jsc.nasa.gov  hpfs-fsl-mds1.fsl.jsc.nasa.gov  
hpfs-fsl-MDT  zfs:mds0-0/meta-fsl

hpfs-fsl-oss00.fsl.jsc.nasa.gov hpfs-fsl-oss01.fsl.jsc.nasa.gov 
hpfs-fsl-OST  zfs:oss00-0/ost-fsl
hpfs-fsl-oss00.fsl.jsc.nasa.gov hpfs-fsl-oss01.fsl.jsc.nasa.gov 
hpfs-fsl-OST000c  zfs:oss00-1/ost-fsl

hpfs-fsl-oss01.fsl.jsc.nasa.gov hpfs-fsl-oss00.fsl.jsc.nasa.gov 
hpfs-fsl-OST0001  zfs:oss01-0/ost-fsl
hpfs-fsl-oss01.fsl.jsc.nasa.gov hpfs-fsl-oss00.fsl.jsc.nasa.gov 
hpfs-fsl-OST000d  zfs:oss01-1/ost-fsl



If you wanted to fail oss01’s OST’s over to oss00, you’d do a “service lustre 
stop” on oss01 followed by a “service lustre start foreign” on oss00.  This 
setup has been stable and has served us well for a long time.  Our servers are 
stable enough that we never set up automated failover via corosync or something 
similar.



From: Vinícius Ferrão 
Date: Sunday, January 7, 2024 at 12:06 PM
To: "Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" 

Cc: Thomas Roth , Lustre Diskussionsliste 

Subject: Re: [lustre-discuss] [EXTERNAL] [BULK] MDS hardware - NVME?

CAUTION: This email originated from outside of NASA.  Please take care when 
clicking links or opening attachments.  Use the "Report Message" button to 
report suspicious messages to the NASA SOC.


Hi Vicker may I ask if you have any kind of HA on this setup?

If yes I’m interested on how the ZFS pools would migrate from one server to 
another in case of failure. I’m considering the typical lustre deployment were 
you have two servers attached to two JBODs using a multipath SAS topology with 
crossed cables: |X|.

I can easily understand that when you have Hardware RAID running on the JBOD 
and SAS HBA on the servers, but for a total software solution I’m unaware how 
that will work effectively.

Thank you.


On 5 Jan 2024, at 14:07, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] 
via lustre-discuss  wrote:

We are in the process of retiring two long standing LFS's (about 8 years old), 
which we built and managed ourselves.  Both use ZFS and have the MDT'S on ssd's 
in a JBOD that require the kind of software-based management you describe, in 
our case ZFS pools built on multipath devices.  The MDT in one is ZFS and the 
MDT in the other LFS is ldiskfs but uses ZFS and a zvol as you describe - we 
build the ldiskfs MDT on top of the zvol.  Generally, this has worked well for 
us, with one big caveat.  If you look for my posts to this list and the ZFS 
list you'll find more details.  The short version is that we utilize ZFS 
snapshots and clones to do backups of the metadata.  We've run into situations 
where the backup process stalls, leaving a clone hanging around.  We've 
experienced a situation a couple of times where the clone and the primary zvol 
get swapped, effectively rolling back our metadata to the point when the clone 
was created.  I have tried, unsuccessfully, to recreate that in a test 
environment.  So if you do that kind of setup, make sure you have good 
monitoring in place to detect if your backups/clones stall.  We've kept up with 
lustre and ZFS updates over the years and are currently on lustre 2.14 and ZFS 
2.1.  We've seen the gap between our ZFS MDT and ldiskfs performance shrink to 
the point where they are pretty much on par to each now.  I think our ZFS MDT 
performance could be better with more hardware and software tuning but our 
small team hasn't had the bandwidth to tackle that.

Our newest LFS is vendor provided and uses NVMe MDT's.  I'm not at liberty to 
talk about the proprietary way those devices are managed.  However, the 
metadata performance is SO much better than our older LFS's, for a lot of 
reasons, but I'd highly recommend NVMe's for your MDT's.

-Original Message-
From: lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>
 > on behalf of Thomas Roth via 
lustre-discuss 
mailto:lustre-discuss@lists.lustre.org> 
>
Reply-To: Thomas Roth mailto:t.r...@gsi.de> 
>
Date: Friday, January 5, 2024 at 9:03 AM
To: Lustre Diskussionsliste 
mailto:lustre-discuss@lists.lustre.org> 
>
Subject: [EXTERNAL] [BULK] 

[lustre-discuss] Extending Lustre file system

[Backer via lustre-discuss]

Hi,

Good morning and happy new year!

I have a quick question on extending a lustre file system. The extension is
performed online. I am looking for any best practices or anything to
watchout while doing the file system extension. The file system extension
is done adding new OSS and many OSTs within these servers.

Really appreciate your help on this.

Regards,
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org