Re: [lustre-discuss] ZFS tuning for MDT/MGS

[Carlson, Timothy S]

+1 on

options zfs zfs_prefetch_disable=1

Might not be as critical now, but that was a must-have on Lustre 2.5.x

Tim

From: lustre-discuss  On Behalf Of 
Riccardo Veraldi
Sent: Wednesday, March 13, 2019 3:00 PM
To: Kurt Strosahl ; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] ZFS tuning for MDT/MGS

these are the zfs settings I use on my MDSes

 zfs set mountpoint=none mdt0
 zfs set sync=disabled mdt0
 zfs set atime=off amdt0
 zfs set redundant_metadata=most mdt0
 zfs set xattr=sa mdt0

if youor MDT partition is on a 4KB sector disk then you can use ashift=12 when 
you create the filesystem but zfs is pretty smart and in my case it recognized 
it automatically and used ashift=12 automatically.

also here are the zfs kernel modules parameters i use to ahve better 
performance. I use it on both MDS and OSSes

options zfs zfs_prefetch_disable=1
options zfs zfs_txg_history=120
options zfs metaslab_debug_unload=1
#
options zfs zfs_vdev_scheduler=deadline
options zfs zfs_vdev_async_write_active_min_dirty_percent=20
#
options zfs zfs_vdev_scrub_min_active=48
options zfs zfs_vdev_scrub_max_active=128
#options zfs zfs_vdev_sync_write_min_active=64
#options zfs zfs_vdev_sync_write_max_active=128
#
options zfs zfs_vdev_sync_write_min_active=8
options zfs zfs_vdev_sync_write_max_active=32
options zfs zfs_vdev_sync_read_min_active=8
options zfs zfs_vdev_sync_read_max_active=32
options zfs zfs_vdev_async_read_min_active=8
options zfs zfs_vdev_async_read_max_active=32
options zfs zfs_top_maxinflight=320
options zfs zfs_txg_timeout=30
options zfs zfs_dirty_data_max_percent=40
options zfs zfs_vdev_async_write_min_active=8
options zfs zfs_vdev_async_write_max_active=32

some people may disagree with me anyway after years of trying different options 
I reached this stable configuration.

then there are a bunch of other important Lustre level optimizations that you 
can do if you are looking for performance increase.

Cheers

Rick

On 3/13/19 11:44 AM, Kurt Strosahl wrote:

Good Afternoon,



I'm reviewing the zfs parameters for a new metadata system and I was 
looking to see if anyone had examples (good or bad) of zfs parameters?  I'm 
assuming that the MDT won't benefit from a recordsize of 1MB, and I've already 
set the ashift to 12.  I'm using an MDT/MGS made up of a stripe across mirrored 
ssds.



w/r,

Kurt



___

lustre-discuss mailing list

lustre-discuss@lists.lustre.org

http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] ZFS tuning for MDT/MGS

[Riccardo Veraldi]

these are the zfs settings I use on my MDSes

 zfs set mountpoint=none mdt0
 zfs set sync=disabled mdt0
 zfs set atime=off amdt0
 zfs set redundant_metadata=most mdt0
 zfs set xattr=sa mdt0

if youor MDT partition is on a 4KB sector disk then you can use 
ashift=12 when you create the filesystem but zfs is pretty smart and in 
my case it recognized it automatically and used ashift=12 automatically.


also here are the zfs kernel modules parameters i use to ahve better 
performance. I use it on both MDS and OSSes


options zfs zfs_prefetch_disable=1
options zfs zfs_txg_history=120
options zfs metaslab_debug_unload=1
#
options zfs zfs_vdev_scheduler=deadline
options zfs zfs_vdev_async_write_active_min_dirty_percent=20
#
options zfs zfs_vdev_scrub_min_active=48
options zfs zfs_vdev_scrub_max_active=128
#options zfs zfs_vdev_sync_write_min_active=64
#options zfs zfs_vdev_sync_write_max_active=128
#
options zfs zfs_vdev_sync_write_min_active=8
options zfs zfs_vdev_sync_write_max_active=32
options zfs zfs_vdev_sync_read_min_active=8
options zfs zfs_vdev_sync_read_max_active=32
options zfs zfs_vdev_async_read_min_active=8
options zfs zfs_vdev_async_read_max_active=32
options zfs zfs_top_maxinflight=320
options zfs zfs_txg_timeout=30
options zfs zfs_dirty_data_max_percent=40
options zfs zfs_vdev_async_write_min_active=8
options zfs zfs_vdev_async_write_max_active=32

some people may disagree with me anyway after years of trying different 
options I reached this stable configuration.


then there are a bunch of other important Lustre level optimizations 
that you can do if you are looking for performance increase.


Cheers

Rick

On 3/13/19 11:44 AM, Kurt Strosahl wrote:


Good Afternoon,


    I'm reviewing the zfs parameters for a new metadata system and I 
was looking to see if anyone had examples (good or bad) of zfs 
parameters? I'm assuming that the MDT won't benefit from a recordsize 
of 1MB, and I've already set the ashift to 12.  I'm using an MDT/MGS 
made up of a stripe across mirrored ssds.



w/r,

Kurt


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] ZFS tuning for MDT/MGS

[Kurt Strosahl]

Good Afternoon,


I'm reviewing the zfs parameters for a new metadata system and I was 
looking to see if anyone had examples (good or bad) of zfs parameters?  I'm 
assuming that the MDT won't benefit from a recordsize of 1MB, and I've already 
set the ashift to 12.  I'm using an MDT/MGS made up of a stripe across mirrored 
ssds.


w/r,

Kurt
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] List of Files on Busted OST

[Paul Edmon]

Excellent.  That should work great.

-Paul Edmon-

On 3/13/19 10:25 AM, Chad DeWitt wrote:

Hi Paul,

lfs findmay do what you want:

lfs find /lustre_mount_point/ --ost /IDs_of_OSTs/
/
/
/IDs_of_OSTs/is comma delimited


Should even grab files that are striped and have a portion of their 
data on the specified OSTS.

/
/
-cd



Chad DeWitt, CISSP

UNC Charlotte *| *ITS – University Research Computing

9201 University City Blvd. *| *Charlotte, NC 28223

ccdew...@uncc.edu  *| *www.uncc.edu




If you are not the intended recipient of this transmission or a person 
responsible for delivering it to the intended recipient, any 
disclosure, copying, distribution, or other use of any of the 
information in this transmission is strictly prohibited. If you have 
received this transmission in error, please notify me immediately by 
reply email or by telephone at 704-687-7802. Thank you.




On Wed, Mar 13, 2019 at 10:18 AM Paul Edmon > wrote:


I have a OSS that is offline.  Is there a way to poll the MDT and
grab a
list of files that are on the affected OST's?  Or is the only method
just scaning the whole filesystem with normal find combined with lfs
getstripe commands?

-Paul Edmon-

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] List of Files on Busted OST

[Chad DeWitt]

Hi Paul,

lfs find may do what you want:

lfs find *lustre_mount_point* --ost *IDs_of_OSTs*

*IDs_of_OSTs* is comma delimited


Should even grab files that are striped and have a portion of their data on
the specified OSTS.

-cd



Chad DeWitt, CISSP

UNC Charlotte *| *ITS – University Research Computing

9201 University City Blvd. *| *Charlotte, NC 28223

ccdew...@uncc.edu *| *www.uncc.edu




If you are not the intended recipient of this transmission or a person
responsible for delivering it to the intended recipient, any disclosure,
copying, distribution, or other use of any of the information in this
transmission is strictly prohibited. If you have received this transmission
in error, please notify me immediately by reply email or by telephone at
704-687-7802. Thank you.


On Wed, Mar 13, 2019 at 10:18 AM Paul Edmon  wrote:

> I have a OSS that is offline.  Is there a way to poll the MDT and grab a
> list of files that are on the affected OST's?  Or is the only method
> just scaning the whole filesystem with normal find combined with lfs
> getstripe commands?
>
> -Paul Edmon-
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] List of Files on Busted OST

[Paul Edmon]

I have a OSS that is offline.  Is there a way to poll the MDT and grab a 
list of files that are on the affected OST's?  Or is the only method 
just scaning the whole filesystem with normal find combined with lfs 
getstripe commands?


-Paul Edmon-

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Crash of a MDS

[Götz Waschk]

Hi everyone,

I'm running a Lustre 2.10.6 file system with two MDS. One of them has
just crashed, I have received these messages:

Message from syslogd@miscmds4 at Mar 13 09:01:24 ...
 kernel:[5584275.388034] LustreError:
321079:0:(lod_object.c:1584:lod_parse_dir_striping()) ASSERTION(
lo->ldo_stripe == ((void *)0) ) failed:

Message from syslogd@miscmds4 at Mar 13 09:01:24 ...
 kernel:[5584275.394146] LustreError:
321079:0:(lod_object.c:1584:lod_parse_dir_striping()) LBUG

Is there a known problem that could cause this?

Regards,
Götz Waschk
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org