Hi Cory, They are both running the same versions on client and mgs server. lfs 2.12.8_6_g5457c37
Not sure if this could be related but our lustre environment started behaving starngly 2/3 weeks ago. Also seeing the below when doing rsync of folders to new destinations. failed: Cannot send after transport endpoint shutdown (108) [root@server1 ~] rm: cannot remove 'Logs: Cannot send after transport endpoint shutdown [root@server1 ~] mv: cannot move /test/lustre/structure1 to '/test/lustre/structure2': Input/output error [root@server1 ~] ~] ll [root@server1 ~] ls: cannot open directory .: Cannot send after transport endpoint shutdown Regards From: Spitz, Cory James <[email protected]> Sent: Monday, 17 October 2022 20:56 To: Cloete, F. (Francois) <[email protected]>; Alastair Basden <[email protected]> Cc: [email protected] Subject: Re: [lustre-discuss] Lustre recycle bin You don't often get email from [email protected]<mailto:[email protected]>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> CAUTION - EXTERNAL SENDER - Please be careful when opening links and attachments. Nedbank - IT Information Security Department (ISD) What version(s) are you using? Do you have an old client and a new-ish server? Very old client versions will disagree with the MDSes about how to clean up objects, resulting in orphans. -Cory On 10/17/22, 3:44 AM, "lustre-discuss" <[email protected]<mailto:[email protected]>> wrote: Thank-you! -----Original Message----- From: Alastair Basden <[email protected]<mailto:[email protected]>> Sent: Monday, 17 October 2022 10:13 To: Cloete, F. (Francois) <[email protected]<mailto:[email protected]>> Cc: Andreas Dilger <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [lustre-discuss] Lustre recycle bin [You don't often get email from [email protected]<mailto:[email protected]>. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- CAUTION - EXTERNAL SENDER - Please be careful when opening links and attachments. Nedbank - IT Information Security Department (ISD) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Hi Francois, We had something similar a few months back - I suspect a bug somewhere. Basically files weren't getting removed from the OST. Eventually, we mounted as ext, and removed them manually, I think. A reboot of the file system meant that rm operations then proceeded correctly after that. Cheers, Alastair. On Mon, 17 Oct 2022, Cloete, F. (Francois) via lustre-discuss wrote: > [EXTERNAL EMAIL] > Hi Andreas, > Our OSTs still display high file-system usage after removing folders. > > Are there any commands that could be run to confirm if the allocated space > which was used by those files have been released successfully ? > > Thanks > Francois > > From: Andreas Dilger <[email protected]<mailto:[email protected]>> > Sent: Saturday, 15 October 2022 00:20 > To: Cloete, F. (Francois) > <[email protected]<mailto:[email protected]>> > Cc: [email protected]<mailto:[email protected]> > Subject: Re: [lustre-discuss] Lustre recycle bin > > You don't often get email from > [email protected]<mailto:[email protected]<mailto:[email protected]%3cmailto:[email protected]>>. > Learn why this is > important<https://aka.ms/LearnAboutSenderIdentification > <https://aka.ms/LearnAboutSenderIdentification%20> > > CAUTION - EXTERNAL SENDER - Please be careful when opening links and > attachments. Nedbank - IT Information Security Department (ISD) There isn't a > recycle bin, but filenames are deleted from the filesystem quickly and the > data objects are deleted in the background asynchronously (with transactions > to prevent the space being leaked). If there are a lot of files this may > take some time, rebooting will not speed it up. > > > On Oct 14, 2022, at 10:00, Cloete, F. (Francois) via lustre-discuss > <[email protected]<mailto:[email protected]<mailto:[email protected]%3cmailto:[email protected]>>> > wrote: > > Hi Community, > Is anyone aware of a recycle bin parameter for Lustre? > > Just deleted a whole lot of files but for some reason the space is not > getting cleared. > > Server rebooted, file-system un-mounted etc. > > Thanks > > ******************** > Nedbank Limited group of companies (Nedbank) disclaimer and confidentiality > notice: > This email, including any attachments (email), contains information that is > confidential and is meant only for intended recipients. You may not share or > copy the email or any part of it, unless the sender has specifically allowed > you to do so. If you are not an intended recipient, please delete the email > permanently and let Nedbank know that you have deleted it by replying to the > sender or calling the Nedbank Contact Centre on 0860 555 111. > This email is not confirmation of a transaction or a Nedbank statement and is > not offering or inviting anyone to take up any financial products or > services, unless the content specifically indicates that it does so. Nedbank > will not be liable for any errors or omissions in this email. The views and > opinions are those of the author and not necessarily those of Nedbank. > The names of the Nedbank Board of Directors and Company Secretary are > available here: http://www.nedbank.co.za/terms/DirectorsNedbank.htm > <http://www.nedbank.co.za/terms/DirectorsNedbank.htm > <http://www.nedbank.co.za/terms/DirectorsNedbank.htm%20> >. Nedbank Ltd Reg > No 1951/000009/06. > ******************** > _______________________________________________ > lustre-discuss mailing list > [email protected]<mailto:[email protected]<mailto:[email protected]%3cmailto:[email protected]> > > > https://eur01.safelinks.protection.outlook.com/?url=http://lists > .lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=05%7C0 > 1%7CFrancoisClo%40Nedbank.co.za%7Caccd70e00cb84df3418608dab017807f%7C0 > b1d23d810d140938cb7fd0bb32f81e1%7C0%7C0%7C638015912276367615%7CUnknown > %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ > XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f4pQ6fAnpWfLFoihgUUGBNXH65%2B9Yc > Wgotw5E7UuUgU%3D&reserved=0 > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Whamcloud > > > > > > > _______________________________________________ lustre-discuss mailing list [email protected]<mailto:[email protected]> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=05%7C01%7CFrancoisClo%40Nedbank.co.za%7Ccd127790e4714b6ffa0908dab0714a6d%7C0b1d23d810d140938cb7fd0bb32f81e1%7C0%7C0%7C638016297920336453%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rHCHfCCnDwn38lq8HE1QeF1a6yaLttJ%2BFnb1B08iEZ0%3D&reserved=0>
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
