Re: [lustre-discuss] ko2iblnd.conf
On the server node(s): options ko2iblnd-opa peer_credits=32 peer_credits_hiw=16 credits=1024 concurrent_sends=64 ntx=2048 map_on_demand=256 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4 On clients: options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4 My concern isn’t so much the mismatch because I know that’s an issue but rather what numbers we should settle on with a recent lustre build. I also see the ko2iblnd-opa in the server config, which means because the server is actually loading ko2iblnd that maybe defaults are used? What made me look was we were seeing lots of: LNetError: 2961324:0:(o2iblnd_cb.c:2612:kiblnd_passive_connect()) Can't accept conn from xxx.xxx.xxx.xxx@o2ib2, queue depth too large: 42 (<=32 wanted) — Dan Szkola FNAL > On Apr 11, 2024, at 12:36 PM, Andreas Dilger wrote: > > [EXTERNAL] – This message is from an external sender > > > On Apr 11, 2024, at 09:56, Daniel Szkola via lustre-discuss > wrote: >> >> Hello all, >> >> I recently discovered some mismatches in our /etc/modprobe.d/ko2iblnd.conf >> files between our clients and servers. >> >> Is it now recommended to keep the defaults on this module and run without a >> config file or are there recommended numbers for lustre-2.15.X? >> >> The only thing I’ve seen that provides any guidance is the Lustre wiki and >> an HP/Cray doc: >> >> https://www.hpe.com/psnow/resources/ebooks/a00113867en_us_v2/Lustre_Server_Recommended_Tuning_Parameters_4.x.html >> >> Anyone have any sage advice on what the ko2iblnd.conf (and possibly >> ko2iblnd-opa.conf and hfi1.conf as well) on modern systems? > > It would be useful to know what specific settings are mismatched. Definitely > some of them need to be consistent between peers, others depend on your > system. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Whamcloud > > > > > > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] ko2iblnd.conf
Hello all, I recently discovered some mismatches in our /etc/modprobe.d/ko2iblnd.conf files between our clients and servers. Is it now recommended to keep the defaults on this module and run without a config file or are there recommended numbers for lustre-2.15.X? The only thing I’ve seen that provides any guidance is the Lustre wiki and an HP/Cray doc: https://www.hpe.com/psnow/resources/ebooks/a00113867en_us_v2/Lustre_Server_Recommended_Tuning_Parameters_4.x.html Anyone have any sage advice on what the ko2iblnd.conf (and possibly ko2iblnd-opa.conf and hfi1.conf as well) on modern systems? — Dan Szkola FNAL ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] [EXTERNAL] [BULK] Re: Ongoing issues with quota
Your help on this issue has been much appreciated, thanks. I deleted all the zero-length files for the group that was having issues. The robinhood report and the quota are now reporting the same number of files. Amazing. Thanks again. — Dan Szkola FNAL > On Oct 18, 2023, at 7:12 AM, Andreas Dilger wrote: > > The zero-length objects are created for the file stripes, but if the MDT > inodes were deleted, but something went wrong with the MDT before the OST > objects were deleted, then the objects would be left behind. > > If the objects are in lost+found with the FID as the filename, then the file > itself is almost certainly already deleted, so fid2path would just return the > file in lost+found. > > I don't think there would be any problem to delete them. > > Cheers, Andreas > >> On Oct 18, 2023, at 08:30, Daniel Szkola wrote: >> >> In this case almost all, if not all, of the files look a lot like this: >> >> -r 1 someuser somegroup 0 Dec 31 1969 >> '[0x200012392:0xe0ad:0x0]-R-0’ >> >> stat shows: >> >> # stat [0x200012392:0xe0ad:0x0]-R-0 >> File: [0x200012392:0xe0ad:0x0]-R-0 >> Size: 0 Blocks: 1 IO Block: 4194304 regular empty file >> Device: a75b4da0h/2807778720dInode: 144116440360870061 Links: 1 >> Access: (0400/-r) Uid: (43667/ someuser) Gid: ( 9349/somegroup) >> Access: 1969-12-31 18:00:00.0 -0600 >> Modify: 1969-12-31 18:00:00.0 -0600 >> Change: 1969-12-31 18:00:00.0 -0600 >> Birth: 2023-01-11 13:01:40.0 -0600 >> >> Not sure what these were or how they ended up in lost+found. I took this >> lustre fs over from folks who have moved on and I’m still trying to wrap my >> head around some of the finer details. In a normal linux fs, usually, not >> always, the blocks will have data in them. These are all zero-length. My >> inclination is to see if I can delete them and be done with it, but I’m a >> bit paranoid. >> >> — >> Dan Szkola >> FNAL >> >> >> >> >> >>> On Oct 17, 2023, at 4:23 PM, Andreas Dilger wrote: >>> >>> The files reported in .lustre/lost+found *ARE* the objects on the OSTs (at >>> least when accessed through a Lustre mountpoint, not if accessed directly >>> on the MDT mounted as ldiskfs), so when they are deleted the space on the >>> OSTs will be freed. >>> >>> As for identification, the OST objects do not have any name information, >>> but they should have UID/GID/PROJID and timestamps that might help >>> identification. >>> >>> Cheers, Andreas >>> >>>>> On Oct 18, 2023, at 03:42, Daniel Szkola wrote: >>>> >>>> OK, so I did find the hidden .lustre directory (thanks Darby) and there >>>> are many, many files in the lost+found directory. I can run ’stat’ on them >>>> and get some info. Is there anything else I can do to tell what these >>>> were? Is it safe to delete them? Is there anyway to tell if there are >>>> matching files on the OST(s) that also need to be deleted? >>>> >>>> — >>>> Dan Szkola >>>> FNAL >>>> >>>>> On Oct 10, 2023, at 3:44 PM, Vicker, Darby J. (JSC-EG111)[Jacobs >>>>> Technology, Inc.] wrote: >>>>> >>>>>> I don’t have a .lustre directory at the filesystem root. >>>>> >>>>> It's there, but doesn't show up even with 'ls -a'. If you cd into it or >>>>> ls it, it's there. Lustre magic. :) >>>>> >>>>> -Original Message- >>>>> From: lustre-discuss >>>> <mailto:lustre-discuss-boun...@lists.lustre.org>> on behalf of Daniel >>>>> Szkola via lustre-discuss >>>> <mailto:lustre-discuss@lists.lustre.org>> >>>>> Reply-To: Daniel Szkola mailto:dszk...@fnal.gov>> >>>>> Date: Tuesday, October 10, 2023 at 2:30 PM >>>>> To: Andreas Dilger mailto:adil...@whamcloud.com>> >>>>> Cc: lustre >>>> <mailto:lustre-discuss@lists.lustre.org>> >>>>> Subject: [EXTERNAL] [BULK] Re: [lustre-discuss] Ongoing issues with quota >>>>> >>>>> >>>>> CAUTION: This email originated from outside of NASA. Please take care >>>>> when clicking links or opening attachments. Use the "Report Message" >>>>> button to report s
Re: [lustre-discuss] [EXTERNAL] [BULK] Re: Ongoing issues with quota
In this case almost all, if not all, of the files look a lot like this: -r 1 someuser somegroup 0 Dec 31 1969 '[0x200012392:0xe0ad:0x0]-R-0’ stat shows: # stat [0x200012392:0xe0ad:0x0]-R-0 File: [0x200012392:0xe0ad:0x0]-R-0 Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: a75b4da0h/2807778720d Inode: 144116440360870061 Links: 1 Access: (0400/-r) Uid: (43667/ someuser) Gid: ( 9349/somegroup) Access: 1969-12-31 18:00:00.0 -0600 Modify: 1969-12-31 18:00:00.0 -0600 Change: 1969-12-31 18:00:00.0 -0600 Birth: 2023-01-11 13:01:40.0 -0600 Not sure what these were or how they ended up in lost+found. I took this lustre fs over from folks who have moved on and I’m still trying to wrap my head around some of the finer details. In a normal linux fs, usually, not always, the blocks will have data in them. These are all zero-length. My inclination is to see if I can delete them and be done with it, but I’m a bit paranoid. — Dan Szkola FNAL > On Oct 17, 2023, at 4:23 PM, Andreas Dilger wrote: > > The files reported in .lustre/lost+found *ARE* the objects on the OSTs (at > least when accessed through a Lustre mountpoint, not if accessed directly on > the MDT mounted as ldiskfs), so when they are deleted the space on the OSTs > will be freed. > > As for identification, the OST objects do not have any name information, but > they should have UID/GID/PROJID and timestamps that might help identification. > > Cheers, Andreas > >> On Oct 18, 2023, at 03:42, Daniel Szkola wrote: >> >> OK, so I did find the hidden .lustre directory (thanks Darby) and there are >> many, many files in the lost+found directory. I can run ’stat’ on them and >> get some info. Is there anything else I can do to tell what these were? Is >> it safe to delete them? Is there anyway to tell if there are matching files >> on the OST(s) that also need to be deleted? >> >> — >> Dan Szkola >> FNAL >> >>> On Oct 10, 2023, at 3:44 PM, Vicker, Darby J. (JSC-EG111)[Jacobs >>> Technology, Inc.] wrote: >>> >>>> I don’t have a .lustre directory at the filesystem root. >>> >>> It's there, but doesn't show up even with 'ls -a'. If you cd into it or ls >>> it, it's there. Lustre magic. :) >>> >>> -Original Message- >>> From: lustre-discuss >> <mailto:lustre-discuss-boun...@lists.lustre.org>> on behalf of Daniel >>> Szkola via lustre-discuss >> <mailto:lustre-discuss@lists.lustre.org>> >>> Reply-To: Daniel Szkola mailto:dszk...@fnal.gov>> >>> Date: Tuesday, October 10, 2023 at 2:30 PM >>> To: Andreas Dilger mailto:adil...@whamcloud.com>> >>> Cc: lustre >> <mailto:lustre-discuss@lists.lustre.org>> >>> Subject: [EXTERNAL] [BULK] Re: [lustre-discuss] Ongoing issues with quota >>> >>> >>> CAUTION: This email originated from outside of NASA. Please take care when >>> clicking links or opening attachments. Use the "Report Message" button to >>> report suspicious messages to the NASA SOC. >>> >>> >>> >>> >>> >>> >>> >>> >>> Hello Andreas, >>> >>> >>> lfs df -i reports 19,204,412 inodes used. When I did the full robinhood >>> scan, it reported scanning 18,673,874 entries, so fairly close. >>> >>> >>> I don’t have a .lustre directory at the filesystem root. >>> >>> >>> Another interesting aspect of this particular issue is I can run lctl lfsck >>> and every time I get: >>> >>> >>> layout_repaired: 1468299 >>> >>> >>> But it doesn’t seem to be actually repairing anything because if I run it >>> again, I’ll get the same or a similar number. >>> >>> >>> I run it like this: >>> lctl lfsck_start -t layout -t namespace -o -M lfsc-MDT >>> >>> >>> — >>> Dan Szkola >>> FNAL >>> >>> >>> >>> >>>> On Oct 10, 2023, at 10:47 AM, Andreas Dilger >>> <mailto:adil...@whamcloud.com>> wrote: >>>> >>>> There is a $ROOT/.lustre/lost+found that you could check. >>>> >>>> What does "lfs df -i" report for the used inode count? Maybe it is RBH >>>> that is reporting the wrong count? >>>> >>>> The other alternative would be to mount the MDT filesystem
Re: [lustre-discuss] [EXTERNAL] [BULK] Re: Ongoing issues with quota
OK, so I did find the hidden .lustre directory (thanks Darby) and there are many, many files in the lost+found directory. I can run ’stat’ on them and get some info. Is there anything else I can do to tell what these were? Is it safe to delete them? Is there anyway to tell if there are matching files on the OST(s) that also need to be deleted? — Dan Szkola FNAL > On Oct 10, 2023, at 3:44 PM, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, > Inc.] wrote: > >> I don’t have a .lustre directory at the filesystem root. > > It's there, but doesn't show up even with 'ls -a'. If you cd into it or ls > it, it's there. Lustre magic. :) > > -Original Message- > From: lustre-discuss <mailto:lustre-discuss-boun...@lists.lustre.org>> on behalf of Daniel Szkola > via lustre-discuss <mailto:lustre-discuss@lists.lustre.org>> > Reply-To: Daniel Szkola mailto:dszk...@fnal.gov>> > Date: Tuesday, October 10, 2023 at 2:30 PM > To: Andreas Dilger mailto:adil...@whamcloud.com>> > Cc: lustre <mailto:lustre-discuss@lists.lustre.org>> > Subject: [EXTERNAL] [BULK] Re: [lustre-discuss] Ongoing issues with quota > > > CAUTION: This email originated from outside of NASA. Please take care when > clicking links or opening attachments. Use the "Report Message" button to > report suspicious messages to the NASA SOC. > > > > > > > > > Hello Andreas, > > > lfs df -i reports 19,204,412 inodes used. When I did the full robinhood scan, > it reported scanning 18,673,874 entries, so fairly close. > > > I don’t have a .lustre directory at the filesystem root. > > > Another interesting aspect of this particular issue is I can run lctl lfsck > and every time I get: > > > layout_repaired: 1468299 > > > But it doesn’t seem to be actually repairing anything because if I run it > again, I’ll get the same or a similar number. > > > I run it like this: > lctl lfsck_start -t layout -t namespace -o -M lfsc-MDT > > > — > Dan Szkola > FNAL > > > > >> On Oct 10, 2023, at 10:47 AM, Andreas Dilger > <mailto:adil...@whamcloud.com>> wrote: >> >> There is a $ROOT/.lustre/lost+found that you could check. >> >> What does "lfs df -i" report for the used inode count? Maybe it is RBH that >> is reporting the wrong count? >> >> The other alternative would be to mount the MDT filesystem directly as type >> ZFS and see what df -i and find report? >> >> Cheers, Andreas >> >>> On Oct 10, 2023, at 22:16, Daniel Szkola via lustre-discuss >>> mailto:lustre-discuss@lists.lustre.org>> >>> wrote: >>> >>> OK, I disabled, waited for a while, then reenabled. I still get the same >>> numbers. The only thing I can think is somehow the count is correct, >>> despite the huge difference. Robinhood and find show about 1.7M files, >>> dirs, and links. The quota is showing a bit over 3.1M inodes used. We only >>> have one MDS and MGS. Any ideas where the discrepancy may lie? Orphans? Is >>> there a lost+found area in lustre? >>> >>> — >>> Dan Szkola >>> FNAL >>> >>> >>>> On Oct 10, 2023, at 8:24 AM, Daniel Szkola >>> <mailto:dszk...@fnal.gov>> wrote: >>>> >>>> Hi Robert, >>>> >>>> Thanks for the response. Do you remember exactly how you did it? Did you >>>> bring everything down at any point? I know you can do this: >>>> >>>> lctl conf_param fsname.quota.mdt=none >>>> >>>> but is that all you did? Did you wait or bring everything down before >>>> reenabling? I’m worried because that allegedly just enables/disables >>>> enforcement and space accounting is always on. Andreas stated that quotas >>>> are controlled by ZFS, but there has been no quota support enabled on any >>>> of the ZFS volumes in our lustre filesystem. >>>> >>>> — >>>> Dan Szkola >>>> FNAL >>>> >>>>>> On Oct 10, 2023, at 2:17 AM, Redl, Robert >>>>> <mailto:robert.r...@lmu.de>> wrote: >>>>> >>>>> Dear Dan, >>>>> >>>>> I had a similar problem some time ago. We are also using ZFS for MDT and >>>>> OSTs. For us, the used disk space was reported wrong. The problem was >>>>> fixed by switching quota support off on the MGS and then on again. >>>>> >&g
Re: [lustre-discuss] Ongoing issues with quota
Hello Andreas, lfs df -i reports 19,204,412 inodes used. When I did the full robinhood scan, it reported scanning 18,673,874 entries, so fairly close. I don’t have a .lustre directory at the filesystem root. Another interesting aspect of this particular issue is I can run lctl lfsck and every time I get: layout_repaired: 1468299 But it doesn’t seem to be actually repairing anything because if I run it again, I’ll get the same or a similar number. I run it like this: lctl lfsck_start -t layout -t namespace -o -M lfsc-MDT — Dan Szkola FNAL > On Oct 10, 2023, at 10:47 AM, Andreas Dilger wrote: > > There is a $ROOT/.lustre/lost+found that you could check. > > What does "lfs df -i" report for the used inode count? Maybe it is RBH that > is reporting the wrong count? > > The other alternative would be to mount the MDT filesystem directly as type > ZFS and see what df -i and find report? > > Cheers, Andreas > >> On Oct 10, 2023, at 22:16, Daniel Szkola via lustre-discuss >> wrote: >> >> OK, I disabled, waited for a while, then reenabled. I still get the same >> numbers. The only thing I can think is somehow the count is correct, despite >> the huge difference. Robinhood and find show about 1.7M files, dirs, and >> links. The quota is showing a bit over 3.1M inodes used. We only have one >> MDS and MGS. Any ideas where the discrepancy may lie? Orphans? Is there a >> lost+found area in lustre? >> >> — >> Dan Szkola >> FNAL >> >> >>> On Oct 10, 2023, at 8:24 AM, Daniel Szkola wrote: >>> >>> Hi Robert, >>> >>> Thanks for the response. Do you remember exactly how you did it? Did you >>> bring everything down at any point? I know you can do this: >>> >>> lctl conf_param fsname.quota.mdt=none >>> >>> but is that all you did? Did you wait or bring everything down before >>> reenabling? I’m worried because that allegedly just enables/disables >>> enforcement and space accounting is always on. Andreas stated that quotas >>> are controlled by ZFS, but there has been no quota support enabled on any >>> of the ZFS volumes in our lustre filesystem. >>> >>> — >>> Dan Szkola >>> FNAL >>> >>>>> On Oct 10, 2023, at 2:17 AM, Redl, Robert wrote: >>>> >>>> Dear Dan, >>>> >>>> I had a similar problem some time ago. We are also using ZFS for MDT and >>>> OSTs. For us, the used disk space was reported wrong. The problem was >>>> fixed by switching quota support off on the MGS and then on again. >>>> >>>> Cheers, >>>> Robert >>>> >>>>> Am 09.10.2023 um 17:55 schrieb Daniel Szkola via lustre-discuss >>>>> : >>>>> >>>>> Thanks, I will look into the ZFS quota since we are using ZFS for all >>>>> storage, MDT and OSTs. >>>>> >>>>> In our case, there is a single MDS/MDT. I have used Robinhood and lfs >>>>> find (by group) commands to verify what the numbers should apparently be. >>>>> >>>>> — >>>>> Dan Szkola >>>>> FNAL >>>>> >>>>>> On Oct 9, 2023, at 10:13 AM, Andreas Dilger >>>>>> wrote: >>>>>> >>>>>> The quota accounting is controlled by the backing filesystem of the OSTs >>>>>> and MDTs. >>>>>> >>>>>> For ldiskfs/ext4 you could run e2fsck to re-count all of the inode and >>>>>> block usage. >>>>>> >>>>>> For ZFS you would have to ask on the ZFS list to see if there is some >>>>>> way to re-count the quota usage. >>>>>> >>>>>> The "inode" quota is accounted from the MDTs, while the "block" quota is >>>>>> accounted from the OSTs. You might be able to see with "lfs quota -v -g >>>>>> group" to see if there is one particular MDT that is returning too many >>>>>> inodes. >>>>>> >>>>>> Possibly if you have directories that are striped across many MDTs it >>>>>> would inflate the used inode count. For example, if every one of the >>>>>> 426k directories reported by RBH was striped across 4 MDTs then you >>>>>> would see the inode count add up to 3.6M. >>>>>> >>>>>> If that was the case, then I
Re: [lustre-discuss] Ongoing issues with quota
OK, I disabled, waited for a while, then reenabled. I still get the same numbers. The only thing I can think is somehow the count is correct, despite the huge difference. Robinhood and find show about 1.7M files, dirs, and links. The quota is showing a bit over 3.1M inodes used. We only have one MDS and MGS. Any ideas where the discrepancy may lie? Orphans? Is there a lost+found area in lustre? — Dan Szkola FNAL > On Oct 10, 2023, at 8:24 AM, Daniel Szkola wrote: > > Hi Robert, > > Thanks for the response. Do you remember exactly how you did it? Did you > bring everything down at any point? I know you can do this: > > lctl conf_param fsname.quota.mdt=none > > but is that all you did? Did you wait or bring everything down before > reenabling? I’m worried because that allegedly just enables/disables > enforcement and space accounting is always on. Andreas stated that quotas are > controlled by ZFS, but there has been no quota support enabled on any of the > ZFS volumes in our lustre filesystem. > > — > Dan Szkola > FNAL > >> On Oct 10, 2023, at 2:17 AM, Redl, Robert wrote: >> >> Dear Dan, >> >> I had a similar problem some time ago. We are also using ZFS for MDT and >> OSTs. For us, the used disk space was reported wrong. The problem was fixed >> by switching quota support off on the MGS and then on again. >> >> Cheers, >> Robert >> >>> Am 09.10.2023 um 17:55 schrieb Daniel Szkola via lustre-discuss >>> : >>> >>> Thanks, I will look into the ZFS quota since we are using ZFS for all >>> storage, MDT and OSTs. >>> >>> In our case, there is a single MDS/MDT. I have used Robinhood and lfs find >>> (by group) commands to verify what the numbers should apparently be. >>> >>> — >>> Dan Szkola >>> FNAL >>> >>>> On Oct 9, 2023, at 10:13 AM, Andreas Dilger wrote: >>>> >>>> The quota accounting is controlled by the backing filesystem of the OSTs >>>> and MDTs. >>>> >>>> For ldiskfs/ext4 you could run e2fsck to re-count all of the inode and >>>> block usage. >>>> >>>> For ZFS you would have to ask on the ZFS list to see if there is some way >>>> to re-count the quota usage. >>>> >>>> The "inode" quota is accounted from the MDTs, while the "block" quota is >>>> accounted from the OSTs. You might be able to see with "lfs quota -v -g >>>> group" to see if there is one particular MDT that is returning too many >>>> inodes. >>>> >>>> Possibly if you have directories that are striped across many MDTs it >>>> would inflate the used inode count. For example, if every one of the 426k >>>> directories reported by RBH was striped across 4 MDTs then you would see >>>> the inode count add up to 3.6M. >>>> >>>> If that was the case, then I would really, really advise against striping >>>> every directory in the filesystem. That will cause problems far worse >>>> than just inflating the inode quota accounting. >>>> >>>> Cheers, Andreas >>>> >>>>> On Oct 9, 2023, at 22:33, Daniel Szkola via lustre-discuss >>>>> wrote: >>>>> >>>>> Is there really no way to force a recount of files used by the quota? >>>>> All indications are we have accounts where files were removed and this is >>>>> not reflected in the used file count in the quota. The space used seems >>>>> correct but the inodes used numbers are way high. There must be a way to >>>>> clear these numbers and have a fresh count done. >>>>> >>>>> — >>>>> Dan Szkola >>>>> FNAL >>>>> >>>>>> On Oct 4, 2023, at 11:37 AM, Daniel Szkola via lustre-discuss >>>>>> wrote: >>>>>> >>>>>> Also, quotas on the OSTS don’t add up to near 3 million files either: >>>>>> >>>>>> [root@lustreclient scratch]# ssh ossnode0 lfs quota -g somegroup -I 0 >>>>>> /lustre1 >>>>>> Disk quotas for grp somegroup (gid 9544): >>>>>> Filesystem kbytes quota limit grace files quota limit >>>>>> grace >>>>>> 1394853459 0 1913344192 - 132863 0 0 >>>>>> - >>>>>> [root@lustreclient scratc
Re: [lustre-discuss] Ongoing issues with quota
Thanks, I will look into the ZFS quota since we are using ZFS for all storage, MDT and OSTs. In our case, there is a single MDS/MDT. I have used Robinhood and lfs find (by group) commands to verify what the numbers should apparently be. — Dan Szkola FNAL > On Oct 9, 2023, at 10:13 AM, Andreas Dilger wrote: > > The quota accounting is controlled by the backing filesystem of the OSTs and > MDTs. > > For ldiskfs/ext4 you could run e2fsck to re-count all of the inode and block > usage. > > For ZFS you would have to ask on the ZFS list to see if there is some way to > re-count the quota usage. > > The "inode" quota is accounted from the MDTs, while the "block" quota is > accounted from the OSTs. You might be able to see with "lfs quota -v -g > group" to see if there is one particular MDT that is returning too many > inodes. > > Possibly if you have directories that are striped across many MDTs it would > inflate the used inode count. For example, if every one of the 426k > directories reported by RBH was striped across 4 MDTs then you would see the > inode count add up to 3.6M. > > If that was the case, then I would really, really advise against striping > every directory in the filesystem. That will cause problems far worse than > just inflating the inode quota accounting. > > Cheers, Andreas > >> On Oct 9, 2023, at 22:33, Daniel Szkola via lustre-discuss >> wrote: >> >> Is there really no way to force a recount of files used by the quota? All >> indications are we have accounts where files were removed and this is not >> reflected in the used file count in the quota. The space used seems correct >> but the inodes used numbers are way high. There must be a way to clear these >> numbers and have a fresh count done. >> >> — >> Dan Szkola >> FNAL >> >>> On Oct 4, 2023, at 11:37 AM, Daniel Szkola via lustre-discuss >>> wrote: >>> >>> Also, quotas on the OSTS don’t add up to near 3 million files either: >>> >>> [root@lustreclient scratch]# ssh ossnode0 lfs quota -g somegroup -I 0 >>> /lustre1 >>> Disk quotas for grp somegroup (gid 9544): >>> Filesystem kbytes quota limit grace files quota limit grace >>> 1394853459 0 1913344192 - 132863 0 0 >>> - >>> [root@lustreclient scratch]# ssh ossnode0 lfs quota -g somegroup -I 1 >>> /lustre1 >>> Disk quotas for grp somegroup (gid 9544): >>> Filesystem kbytes quota limit grace files quota limit grace >>> 1411579601 0 1963246413 - 120643 0 0 >>> - >>> [root@lustreclient scratch]# ssh ossnode1 lfs quota -g somegroup -I 2 >>> /lustre1 >>> Disk quotas for grp somegroup (gid 9544): >>> Filesystem kbytes quota limit grace files quota limit grace >>> 1416507527 0 1789950778 - 190687 0 0 >>> - >>> [root@lustreclient scratch]# ssh ossnode1 lfs quota -g somegroup -I 3 >>> /lustre1 >>> Disk quotas for grp somegroup (gid 9544): >>> Filesystem kbytes quota limit grace files quota limit grace >>> 1636465724 0 1926578117 - 195034 0 0 >>> - >>> [root@lustreclient scratch]# ssh ossnode2 lfs quota -g somegroup -I 4 >>> /lustre1 >>> Disk quotas for grp somegroup (gid 9544): >>> Filesystem kbytes quota limit grace files quota limit grace >>> 2202272244 0 3020159313 - 185097 0 0 >>> - >>> [root@lustreclient scratch]# ssh ossnode2 lfs quota -g somegroup -I 5 >>> /lustre1 >>> Disk quotas for grp somegroup (gid 9544): >>> Filesystem kbytes quota limit grace files quota limit grace >>> 1324770165 0 1371244768 - 145347 0 0 >>> - >>> [root@lustreclient scratch]# ssh ossnode3 lfs quota -g somegroup -I 6 >>> /lustre1 >>> Disk quotas for grp somegroup (gid 9544): >>> Filesystem kbytes quota limit grace files quota limit grace >>> 2892027349 0 3221225472 - 169386 0 0 >>> - >>> [root@lustreclient scratch]# ssh ossnode3 lfs quota -g somegroup -I 7 >>> /lustre1 >>> Disk quotas for grp somegroup (gid 9544): >>> Filesystem kbytes quota limit grace files quota limit grace >
Re: [lustre-discuss] Ongoing issues with quota
Is there really no way to force a recount of files used by the quota? All indications are we have accounts where files were removed and this is not reflected in the used file count in the quota. The space used seems correct but the inodes used numbers are way high. There must be a way to clear these numbers and have a fresh count done. — Dan Szkola FNAL > On Oct 4, 2023, at 11:37 AM, Daniel Szkola via lustre-discuss > wrote: > > Also, quotas on the OSTS don’t add up to near 3 million files either: > > [root@lustreclient scratch]# ssh ossnode0 lfs quota -g somegroup -I 0 /lustre1 > Disk quotas for grp somegroup (gid 9544): > Filesystem kbytes quota limit grace files quota limit grace >1394853459 0 1913344192 - 132863 0 0 > - > [root@lustreclient scratch]# ssh ossnode0 lfs quota -g somegroup -I 1 /lustre1 > Disk quotas for grp somegroup (gid 9544): > Filesystem kbytes quota limit grace files quota limit grace >1411579601 0 1963246413 - 120643 0 0 > - > [root@lustreclient scratch]# ssh ossnode1 lfs quota -g somegroup -I 2 /lustre1 > Disk quotas for grp somegroup (gid 9544): > Filesystem kbytes quota limit grace files quota limit grace >1416507527 0 1789950778 - 190687 0 0 > - > [root@lustreclient scratch]# ssh ossnode1 lfs quota -g somegroup -I 3 /lustre1 > Disk quotas for grp somegroup (gid 9544): > Filesystem kbytes quota limit grace files quota limit grace >1636465724 0 1926578117 - 195034 0 0 > - > [root@lustreclient scratch]# ssh ossnode2 lfs quota -g somegroup -I 4 /lustre1 > Disk quotas for grp somegroup (gid 9544): > Filesystem kbytes quota limit grace files quota limit grace >2202272244 0 3020159313 - 185097 0 0 > - > [root@lustreclient scratch]# ssh ossnode2 lfs quota -g somegroup -I 5 /lustre1 > Disk quotas for grp somegroup (gid 9544): > Filesystem kbytes quota limit grace files quota limit grace >1324770165 0 1371244768 - 145347 0 0 > - > [root@lustreclient scratch]# ssh ossnode3 lfs quota -g somegroup -I 6 /lustre1 > Disk quotas for grp somegroup (gid 9544): > Filesystem kbytes quota limit grace files quota limit grace >2892027349 0 3221225472 - 169386 0 0 > - > [root@lustreclient scratch]# ssh ossnode3 lfs quota -g somegroup -I 7 /lustre1 > Disk quotas for grp somegroup (gid 9544): > Filesystem kbytes quota limit grace files quota limit grace >2076201636 0 2474853207 - 171552 0 0 > - > > > — > Dan Szkola > FNAL > >> On Oct 4, 2023, at 8:45 AM, Daniel Szkola via lustre-discuss >> wrote: >> >> No combination of ossnodek runs has helped with this. >> >> Again, robinhood shows 1796104 files for the group, an 'lfs find -G gid' >> found 1796104 files as well. >> >> So why is the quota command showing over 3 million inodes used? >> >> There must be a way to force it to recount or clear all stale quota data and >> have it regenerate it? >> >> Anyone? >> >> — >> Dan Szkola >> FNAL >> >> >>> On Sep 27, 2023, at 9:42 AM, Daniel Szkola via lustre-discuss >>> wrote: >>> >>> We have a lustre filesystem that we just upgraded to 2.15.3, however this >>> problem has been going on for some time. >>> >>> The quota command shows this: >>> >>> Disk quotas for grp somegroup (gid 9544): >>> Filesystemused quota limit grace files quota limit grace >>> /lustre1 13.38T 40T 45T - 3136761* 2621440 3670016 >>> expired >>> >>> The group is not using nearly that many files. We have robinhood installed >>> and it show this: >>> >>> Using config file '/etc/robinhood.d/lustre1.conf'. >>> group, type, count, volume, spc_used, avg_size >>> somegroup, symlink, 59071,5.12 MB, 103.16 MB, 91 >>> somegroup, dir, 426619,5.24 GB,5.24 GB, 12.87 KB >>> somegroup, file,1310414, 16.24 TB, 13.37 TB, 13.00 MB >>> >>> Total: 1796104 entries, volume: 17866508365925 bytes (16.25 TB), space >>> used: 14704924899840 bytes (13.37
Re: [lustre-discuss] Ongoing issues with quota
Also, quotas on the OSTS don’t add up to near 3 million files either: [root@lustreclient scratch]# ssh ossnode0 lfs quota -g somegroup -I 0 /lustre1 Disk quotas for grp somegroup (gid 9544): Filesystem kbytes quota limit grace files quota limit grace 1394853459 0 1913344192 - 132863 0 0 - [root@lustreclient scratch]# ssh ossnode0 lfs quota -g somegroup -I 1 /lustre1 Disk quotas for grp somegroup (gid 9544): Filesystem kbytes quota limit grace files quota limit grace 1411579601 0 1963246413 - 120643 0 0 - [root@lustreclient scratch]# ssh ossnode1 lfs quota -g somegroup -I 2 /lustre1 Disk quotas for grp somegroup (gid 9544): Filesystem kbytes quota limit grace files quota limit grace 1416507527 0 1789950778 - 190687 0 0 - [root@lustreclient scratch]# ssh ossnode1 lfs quota -g somegroup -I 3 /lustre1 Disk quotas for grp somegroup (gid 9544): Filesystem kbytes quota limit grace files quota limit grace 1636465724 0 1926578117 - 195034 0 0 - [root@lustreclient scratch]# ssh ossnode2 lfs quota -g somegroup -I 4 /lustre1 Disk quotas for grp somegroup (gid 9544): Filesystem kbytes quota limit grace files quota limit grace 2202272244 0 3020159313 - 185097 0 0 - [root@lustreclient scratch]# ssh ossnode2 lfs quota -g somegroup -I 5 /lustre1 Disk quotas for grp somegroup (gid 9544): Filesystem kbytes quota limit grace files quota limit grace 1324770165 0 1371244768 - 145347 0 0 - [root@lustreclient scratch]# ssh ossnode3 lfs quota -g somegroup -I 6 /lustre1 Disk quotas for grp somegroup (gid 9544): Filesystem kbytes quota limit grace files quota limit grace 2892027349 0 3221225472 - 169386 0 0 - [root@lustreclient scratch]# ssh ossnode3 lfs quota -g somegroup -I 7 /lustre1 Disk quotas for grp somegroup (gid 9544): Filesystem kbytes quota limit grace files quota limit grace 2076201636 0 2474853207 - 171552 0 0 - — Dan Szkola FNAL > On Oct 4, 2023, at 8:45 AM, Daniel Szkola via lustre-discuss > wrote: > > No combination of ossnodek runs has helped with this. > > Again, robinhood shows 1796104 files for the group, an 'lfs find -G gid' > found 1796104 files as well. > > So why is the quota command showing over 3 million inodes used? > > There must be a way to force it to recount or clear all stale quota data and > have it regenerate it? > > Anyone? > > — > Dan Szkola > FNAL > > >> On Sep 27, 2023, at 9:42 AM, Daniel Szkola via lustre-discuss >> wrote: >> >> We have a lustre filesystem that we just upgraded to 2.15.3, however this >> problem has been going on for some time. >> >> The quota command shows this: >> >> Disk quotas for grp somegroup (gid 9544): >>Filesystemused quota limit grace files quota limit grace >> /lustre1 13.38T 40T 45T - 3136761* 2621440 3670016 >> expired >> >> The group is not using nearly that many files. We have robinhood installed >> and it show this: >> >> Using config file '/etc/robinhood.d/lustre1.conf'. >>group, type, count, volume, spc_used, avg_size >> somegroup, symlink, 59071,5.12 MB, 103.16 MB, 91 >> somegroup, dir, 426619,5.24 GB,5.24 GB, 12.87 KB >> somegroup, file,1310414, 16.24 TB, 13.37 TB, 13.00 MB >> >> Total: 1796104 entries, volume: 17866508365925 bytes (16.25 TB), space used: >> 14704924899840 bytes (13.37 TB) >> >> Any ideas what is wrong here? >> >> — >> Dan Szkola >> FNAL >> ___ >> lustre-discuss mailing list >> lustre-discuss@lists.lustre.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwIGaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=e9DXjyTaQ786Tg7WH7oIVaQOA1YDRqyxHOUaYU2_LQw&m=Nk1MkSBTpT-KnrXzEvOOP5tZoVAKyHfPvB-o8_OhewuwHF6S0KelH_WPMLq8IRnR&s=JzAV0C2_CqaDUOG0wZr0mx5tiblBde6ZRUuIHZ2n9DI&e= >> > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwIGaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=e9DXjyTaQ786Tg7WH7oIVaQOA1YDRqyxHOUaYU2_LQw&m=k8TeSgok6MIb-uQMJaquDJS0FQPt0RQxysFNe4d7Rp5TMqGtcqdlezA_TZNuoTJS&s=SRDKhUKQgMW9_OohjyrkzKNYbzTw_M5BJk-bmEi_6w4&e= > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Ongoing issues with quota
Hi Mark, All nodes are using ZFS. OSTs, MDT, and MGT are all ZFS-based, so there's really no way to fsck them. I could do a scrub, but that's not the same thing. Is there a Lustre/ZFS equivalent of 'tune2fs -O [^]quota' for ZFS? I'm guessing that at some point, a large number of files was removed and somehow quota accounting missed this. There should be a simple way to reconcile or regenerate what quota has recorded vs what is actually on disk, which I have verified two different ways. -- Dan On Wed, 2023-10-04 at 15:01 +0100, Mark Dixon wrote: > Hi Dan, > > I think it gets corrected when you umount and fsck the OST's themselves > (not lfsck). At least I recall seeing such messages when fsck'ing on 2.12. > > Best, > > Mark > > On Wed, 4 Oct 2023, Daniel Szkola via lustre-discuss wrote: > > > [EXTERNAL EMAIL] > > > > No combination of lfsck runs has helped with this. > > > > Again, robinhood shows 1796104 files for the group, an 'lfs find -G gid' > > found 1796104 files as well. > > > > So why is the quota command showing over 3 million inodes used? > > > > There must be a way to force it to recount or clear all stale quota data > > and have it regenerate it? > > > > Anyone? > > > > — > > Dan Szkola > > FNAL > > > > > > > On Sep 27, 2023, at 9:42 AM, Daniel Szkola via lustre-discuss > > > wrote: > > > > > > We have a lustre filesystem that we just upgraded to 2.15.3, however > > > this problem has been going on for some time. > > > > > > The quota command shows this: > > > > > > Disk quotas for grp somegroup (gid 9544): > > > Filesystem used quota limit grace files quota > > > limit grace > > > /lustre1 13.38T 40T 45T - 3136761* 2621440 > > > 3670016 expired > > > > > > The group is not using nearly that many files. We have robinhood > > > installed and it show this: > > > > > > Using config file '/etc/robinhood.d/lustre1.conf'. > > > group, type, count, volume, spc_used, avg_size > > > somegroup, symlink, 59071, 5.12 MB, 103.16 MB, 91 > > > somegroup, dir, 426619, 5.24 GB, 5.24 GB, 12.87 KB > > > somegroup, file, 1310414, 16.24 TB, 13.37 TB, 13.00 MB > > > > > > Total: 1796104 entries, volume: 17866508365925 bytes (16.25 TB), space > > > used: 14704924899840 bytes (13.37 TB) > > > > > > Any ideas what is wrong here? > > > > > > — > > > Dan Szkola > > > FNAL > > > ___ > > > lustre-discuss mailing list > > > lustre-discuss@lists.lustre.org > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwIGaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=e9DXjyTaQ786Tg7WH7oIVaQOA1YDRqyxHOUaYU2_LQw&m=Nk1MkSBTpT-KnrXzEvOOP5tZoVAKyHfPvB-o8_OhewuwHF6S0KelH_WPMLq8IRnR&s=JzAV0C2_CqaDUOG0wZr0mx5tiblBde6ZRUuIHZ2n9DI&e= > > > > ___ > > lustre-discuss mailing list > > lustre-discuss@lists.lustre.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwIDaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=e9DXjyTaQ786Tg7WH7oIVaQOA1YDRqyxHOUaYU2_LQw&m=BBVt50ufoqbL64MfSKVa87fK1B4Q0n91KVNJVmvb-9q9xOYwnzpZcOXWgUeM6fxQ&s=uTJ98MgxxcM61HIDJRBpfJpuLDt9Ug4ARh8P_Api3xQ&e= > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Ongoing issues with quota
No combination of lfsck runs has helped with this. Again, robinhood shows 1796104 files for the group, an 'lfs find -G gid' found 1796104 files as well. So why is the quota command showing over 3 million inodes used? There must be a way to force it to recount or clear all stale quota data and have it regenerate it? Anyone? — Dan Szkola FNAL > On Sep 27, 2023, at 9:42 AM, Daniel Szkola via lustre-discuss > wrote: > > We have a lustre filesystem that we just upgraded to 2.15.3, however this > problem has been going on for some time. > > The quota command shows this: > > Disk quotas for grp somegroup (gid 9544): > Filesystemused quota limit grace files quota limit grace > /lustre1 13.38T 40T 45T - 3136761* 2621440 3670016 > expired > > The group is not using nearly that many files. We have robinhood installed > and it show this: > > Using config file '/etc/robinhood.d/lustre1.conf'. > group, type, count, volume, spc_used, avg_size > somegroup, symlink, 59071,5.12 MB, 103.16 MB, 91 > somegroup, dir, 426619,5.24 GB,5.24 GB, 12.87 KB > somegroup, file,1310414, 16.24 TB, 13.37 TB, 13.00 MB > > Total: 1796104 entries, volume: 17866508365925 bytes (16.25 TB), space used: > 14704924899840 bytes (13.37 TB) > > Any ideas what is wrong here? > > — > Dan Szkola > FNAL > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwIGaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=e9DXjyTaQ786Tg7WH7oIVaQOA1YDRqyxHOUaYU2_LQw&m=Nk1MkSBTpT-KnrXzEvOOP5tZoVAKyHfPvB-o8_OhewuwHF6S0KelH_WPMLq8IRnR&s=JzAV0C2_CqaDUOG0wZr0mx5tiblBde6ZRUuIHZ2n9DI&e= > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Ongoing issues with quota
We have a lustre filesystem that we just upgraded to 2.15.3, however this problem has been going on for some time. The quota command shows this: Disk quotas for grp somegroup (gid 9544): Filesystemused quota limit grace files quota limit grace /lustre1 13.38T 40T 45T - 3136761* 2621440 3670016 expired The group is not using nearly that many files. We have robinhood installed and it show this: Using config file '/etc/robinhood.d/lustre1.conf'. group, type, count, volume, spc_used, avg_size somegroup, symlink, 59071,5.12 MB, 103.16 MB, 91 somegroup, dir, 426619,5.24 GB,5.24 GB, 12.87 KB somegroup, file,1310414, 16.24 TB, 13.37 TB, 13.00 MB Total: 1796104 entries, volume: 17866508365925 bytes (16.25 TB), space used: 14704924899840 bytes (13.37 TB) Any ideas what is wrong here? — Dan Szkola FNAL ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OST not freeing space for deleted files?
I’m not seeing anything obvious. Today, the inode counts are increased and the group has reached their hard limit. We have Robinhood running and the numbers there seem accurate but the quota numbers are still high. I’m seeing things like this on the MDS node in dmesg: [Wed Jan 11 11:39:07 2023] LustreError: 39308:0:(osp_dev.c:1682:osp_iocontrol()) lfsc-OST0004-osc-MDT: unrecognized ioctl 0xc00866e6 by lctl [Wed Jan 11 11:39:14 2023] LustreError: 39314:0:(class_obd.c:465:class_handle_ioctl()) OBD ioctl : No Device -12066 [Wed Jan 11 11:39:38 2023] LustreError: 39385:0:(class_obd.c:465:class_handle_ioctl()) OBD ioctl : No Device -12066 [Wed Jan 11 11:39:38 2023] LustreError: 39385:0:(class_obd.c:465:class_handle_ioctl()) Skipped 1 previous similar message [Wed Jan 11 12:06:12 2023] LustreError: 41360:0:(lod_dev.c:1551:lod_sync()) lfsc-MDT-mdtlov: can't sync ost 4: rc = -110 [Wed Jan 11 12:06:12 2023] LustreError: 41360:0:(lod_dev.c:1551:lod_sync()) Skipped 1 previous similar message [Wed Jan 11 12:09:30 2023] LustreError: 41362:0:(lod_dev.c:1551:lod_sync()) lfsc-MDT-mdtlov: can't sync ost 4: rc = -110 [Wed Jan 11 16:18:27 2023] LustreError: 41360:0:(lod_dev.c:1551:lod_sync()) lfsc-MDT-mdtlov: can't sync ost 4: rc = -110 Only seeing this for OST4 though and not 5, both of which seem to be having the problem. So, these may be harmless. I still don’t know why the destroys_in_flight are over 13 million and not decreasing. Any ideas? — Dan Szkola FNAL > On Jan 12, 2023, at 2:59 AM, Degremont, Aurelien wrote: > > Hello Daniel, > > You should also check if there is not some user workload that is triggering > that load, like a constant load of SYNC to files on those OSTs by example. > > Aurélien > > Le 11/01/2023 22:37, « lustre-discuss au nom de Daniel Szkola via > lustre-discuss » lustre-discuss@lists.lustre.org> a écrit : > >CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > >We recently had to take an OSS node that hosts two OSTs out of service to > test the hardware as it was randomly power cycling. > >I migrated all files off of the two OSTs and after some testing we brought > the node back into service after recreating the ZFS pools >and the two OSTs. Since then it’s been mostly working fine, however we’ve > noticed a few group quotas reporting file usage that doesn’t >seem to match what is actually on the filesystem. The inode counts seem to > be correct, but the space used is way too high. > >After lots of poking around I am seeing this on the two OSTS: > >osp.lfsc-OST0004-osc-MDT.sync_changes=13802381 >osp.lfsc-OST0005-osc-MDT.sync_changes=13060667 > >I upped the max_rpcs_in_progress and max_rpcs_in_flight for the two OSTs, > but that just caused the numbers to dip slightly. >All other OSTs have 0 for that value. Also destroys_in_flight show similar > numbers for the two OSTs. > >Any ideas how I can remedy this? > >Lustre 2.12.8 >ZFS 0.7.13 > >— >Dan Szkola >FNAL > > > > > >___ >lustre-discuss mailing list >lustre-discuss@lists.lustre.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwIGaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=e9DXjyTaQ786Tg7WH7oIVaQOA1YDRqyxHOUaYU2_LQw&m=DBtSEnlwRKd7IUYAtj21XR88qwWp8PCksiUQy7Mn0imnzYiq8OhdYUVdjx3aGoyR&s=T29TaXoWSYBTh5eRNhMflhEe2YEQu8M1CDqrp_NSNMg&e= > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] OST not freeing space for deleted files?
We recently had to take an OSS node that hosts two OSTs out of service to test the hardware as it was randomly power cycling. I migrated all files off of the two OSTs and after some testing we brought the node back into service after recreating the ZFS pools and the two OSTs. Since then it’s been mostly working fine, however we’ve noticed a few group quotas reporting file usage that doesn’t seem to match what is actually on the filesystem. The inode counts seem to be correct, but the space used is way too high. After lots of poking around I am seeing this on the two OSTS: osp.lfsc-OST0004-osc-MDT.sync_changes=13802381 osp.lfsc-OST0005-osc-MDT.sync_changes=13060667 I upped the max_rpcs_in_progress and max_rpcs_in_flight for the two OSTs, but that just caused the numbers to dip slightly. All other OSTs have 0 for that value. Also destroys_in_flight show similar numbers for the two OSTs. Any ideas how I can remedy this? Lustre 2.12.8 ZFS 0.7.13 — Dan Szkola FNAL ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Quota issue after OST removal
I did show a 'lfs quota -g somegroup' in the original post and yes, each OST is at the limit that was originally allocated, especially after migrating the files off of the two OSTs before removal. However, I think you may be misreading the issue here. The total quota is 27T and all the files on the remaining OSTs adds up to just over 21T because two OSTs have been removed permanently. The permanently removed OSTs should not be part of the calculations anymore. When the two OSTs were removed, shouldn't the quota be split among the remaining OSTs with each OST given a bigger share of the overall quota? Is there a way to force this? Will restarting the MDS cause this to happen? I just changed the soft/hard limits to 37T/40T from 27T/30T and that does allocate more space per OST, but putting it back to 27T/30T puts the original values back and the group is again shown as exceeding quota. Why is setquota still using the removed OSTs? You can see in the listing where it is still looking for ost4 and ost5. quotactl ost4 failed. quotactl ost5 failed. -- Dan Szkola FNAL On Wed, 2022-10-26 at 21:00 +0200, Thomas Roth via lustre-discuss wrote: > Hi Daniel, > > isn't this expected: on your lustrefs-OST0001, usage seems to have hit > the limit (perhaps if you do 'lfs quota -g somegroup...', it will show > you by how many bytes). > > If one part of the distributed quota is exceeded, Lustre should report > that with the * - although the total across the file system is still below > the > limit. > > > Obviously your 'somegroup' is at the quota limit on all visible OSTs, so > my guess is that would be the same on the missing two OSTs. > So, either have some data removed or increase the limit. > > Best regards > Thomas > > On 26.10.22 16:52, Daniel Szkola via lustre-discuss wrote: > > Hello all, > > > > We recently removed an OSS/OST node that was spontaneously shutting down > > so > > hardware testing could be performed. I have no idea how long it will be > > out, > > so I followed the procedure for permanent removal. > > > > Since then space usage is being calculated correctly, but 'lfs quota' > > will > > show groups as exceeding quota, despite being under both soft and hard > > limits. A verbose listing shows that all OST limits are met and I have > > no > > idea how to reset the limits now that the two OSTs on the removed OSS > > node > > are not part of the equation. > > > > Due to the heavy usage of the Lustre filesystem, no clients have been > > unmounted and no MDS or OST nodes have been restarted. The underlying > > filesystem is ZFS. > > > > Looking for ideas on how to correct this. > > > > Example: > > > > # lfs quota -gh somegroup -v /lustre1 > > Disk quotas for grp somegroup (gid ): > > Filesystem used quota limit grace files quota limit > > grace > > /lustre1 21.59T* 27T 30T 6d23h39m15s 2250592 2621440 > > 3145728 > > - > > lustrefs-MDT_UUID > > 1.961G - 1.962G - 2250592 - 2359296 > > - > > lustrefs-OST_UUID > > 2.876T - 2.876T - - - - > > - > > lustrefs-OST0001_UUID > > 2.611T* - 2.611T - - - - > > - > > lustrefs-OST0002_UUID > > 4.794T - 4.794T - - - - > > - > > lustrefs-OST0003_UUID > > 4.587T - 4.587T - - - - > > - > > quotactl ost4 failed. > > quotactl ost5 failed. > > lustrefs-OST0006_UUID > > 3.21T - 3.21T - - - - > > - > > lustrefs-OST0007_UUID > > 3.515T - 3.515T - - - - > > - > > Total allocated inode limit: 2359296, total allocated block limit: > > 21.59T > > Some errors happened when getting quota info. Some devices may be not > > working or deactivated. The data in "[]" is inaccurate. > > > > -- > > Dan Szkola > > FNAL > > ___ > > lustre-discuss mailing list > > lustre-discuss@lists.lustre.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwICAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=e9DXjyTaQ786Tg7WH7oIVaQOA1YDRqyxHOUaYU2_LQw&m=5r-NvoGFgbr_fVjvWQhbhz8QVqYg9ZRVU5EEQik2CBRsJg5sReIlz9B-UX7VHKey&s=ymWPMGZy5p2pXtOBT_qsRHVMNT5OGmAACZpJj9xUq0k&e= > > > ___
[lustre-discuss] Quota issue after OST removal
Hello all, We recently removed an OSS/OST node that was spontaneously shutting down so hardware testing could be performed. I have no idea how long it will be out, so I followed the procedure for permanent removal. Since then space usage is being calculated correctly, but 'lfs quota' will show groups as exceeding quota, despite being under both soft and hard limits. A verbose listing shows that all OST limits are met and I have no idea how to reset the limits now that the two OSTs on the removed OSS node are not part of the equation. Due to the heavy usage of the Lustre filesystem, no clients have been unmounted and no MDS or OST nodes have been restarted. The underlying filesystem is ZFS. Looking for ideas on how to correct this. Example: # lfs quota -gh somegroup -v /lustre1 Disk quotas for grp somegroup (gid ): Filesystemused quota limit grace files quota limit grace /lustre1 21.59T*27T 30T 6d23h39m15s 2250592 2621440 3145728 - lustrefs-MDT_UUID 1.961G - 1.962G - 2250592 - 2359296 - lustrefs-OST_UUID 2.876T - 2.876T - - - - - lustrefs-OST0001_UUID 2.611T* - 2.611T - - - - - lustrefs-OST0002_UUID 4.794T - 4.794T - - - - - lustrefs-OST0003_UUID 4.587T - 4.587T - - - - - quotactl ost4 failed. quotactl ost5 failed. lustrefs-OST0006_UUID 3.21T - 3.21T - - - - - lustrefs-OST0007_UUID 3.515T - 3.515T - - - - - Total allocated inode limit: 2359296, total allocated block limit: 21.59T Some errors happened when getting quota info. Some devices may be not working or deactivated. The data in "[]" is inaccurate. -- Dan Szkola FNAL ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
