Hi François [root@server1 ~] rm: cannot remove ‘Logs: Cannot send after transport endpoint shutdown [root@server1 ~] mv: cannot move /test/lustre/structure1 to ‘/test/lustre/structure2’: Input/output error
These 2 error messages are typical from a client eviction issue. Your client was evicted (likely from the MDT) and you should expect ESHUTDOWN or EIO in that situation. Look at the kernel logs (dmesg) from both client and server to try understand. The client usually reconnect quickly when this happens and following access should not return error. Aurélien De : lustre-discuss <lustre-discuss-boun...@lists.lustre.org> au nom de "Cloete, F. (Francois) via lustre-discuss" <lustre-discuss@lists.lustre.org> Répondre à : "Cloete, F. (Francois)" <francois...@nedbank.co.za> Date : jeudi 20 octobre 2022 à 08:48 À : "Spitz, Cory James" <cory.sp...@hpe.com>, Alastair Basden <a.g.bas...@durham.ac.uk> Cc : "lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org> Objet : RE: [EXTERNAL][lustre-discuss] Lustre recycle bin CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi Cory, They are both running the same versions on client and mgs server. lfs 2.12.8_6_g5457c37 Not sure if this could be related but our lustre environment started behaving starngly 2/3 weeks ago. Also seeing the below when doing rsync of folders to new destinations. failed: Cannot send after transport endpoint shutdown (108) [root@server1 ~] rm: cannot remove ‘Logs: Cannot send after transport endpoint shutdown [root@server1 ~] mv: cannot move /test/lustre/structure1 to ‘/test/lustre/structure2’: Input/output error [root@server1 ~] ~] ll [root@server1 ~] ls: cannot open directory .: Cannot send after transport endpoint shutdown Regards From: Spitz, Cory James <cory.sp...@hpe.com> Sent: Monday, 17 October 2022 20:56 To: Cloete, F. (Francois) <francois...@nedbank.co.za>; Alastair Basden <a.g.bas...@durham.ac.uk> Cc: lustre-discuss@lists.lustre.org Subject: Re: [lustre-discuss] Lustre recycle bin You don't often get email from cory.sp...@hpe.com<mailto:cory.sp...@hpe.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> CAUTION - EXTERNAL SENDER - Please be careful when opening links and attachments. Nedbank - IT Information Security Department (ISD) What version(s) are you using? Do you have an old client and a new-ish server? Very old client versions will disagree with the MDSes about how to clean up objects, resulting in orphans. -Cory On 10/17/22, 3:44 AM, "lustre-discuss" <lustre-discuss-boun...@lists.lustre.org<mailto:lustre-discuss-boun...@lists.lustre.org>> wrote: Thank-you! -----Original Message----- From: Alastair Basden <a.g.bas...@durham.ac.uk<mailto:a.g.bas...@durham.ac.uk>> Sent: Monday, 17 October 2022 10:13 To: Cloete, F. (Francois) <francois...@nedbank.co.za<mailto:francois...@nedbank.co.za>> Cc: Andreas Dilger <adil...@whamcloud.com<mailto:adil...@whamcloud.com>>; lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> Subject: Re: [lustre-discuss] Lustre recycle bin [You don't often get email from a.g.bas...@durham.ac.uk<mailto:a.g.bas...@durham.ac.uk>. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- CAUTION - EXTERNAL SENDER - Please be careful when opening links and attachments. Nedbank - IT Information Security Department (ISD) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Hi Francois, We had something similar a few months back - I suspect a bug somewhere. Basically files weren't getting removed from the OST. Eventually, we mounted as ext, and removed them manually, I think. A reboot of the file system meant that rm operations then proceeded correctly after that. Cheers, Alastair. On Mon, 17 Oct 2022, Cloete, F. (Francois) via lustre-discuss wrote: > [EXTERNAL EMAIL] > Hi Andreas, > Our OSTs still display high file-system usage after removing folders. > > Are there any commands that could be run to confirm if the allocated space > which was used by those files have been released successfully ? > > Thanks > Francois > > From: Andreas Dilger <adil...@whamcloud.com<mailto:adil...@whamcloud.com>> > Sent: Saturday, 15 October 2022 00:20 > To: Cloete, F. (Francois) > <francois...@nedbank.co.za<mailto:francois...@nedbank.co.za>> > Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> > Subject: Re: [lustre-discuss] Lustre recycle bin > > You don't often get email from > adil...@whamcloud.com<mailto:adil...@whamcloud.com<mailto:adil...@whamcloud.com%3cmailto:adil...@whamcloud.com>>. > Learn why this is > important<https://aka.ms/LearnAboutSenderIdentification > <https://aka.ms/LearnAboutSenderIdentification%20> > > CAUTION - EXTERNAL SENDER - Please be careful when opening links and > attachments. Nedbank - IT Information Security Department (ISD) There isn't a > recycle bin, but filenames are deleted from the filesystem quickly and the > data objects are deleted in the background asynchronously (with transactions > to prevent the space being leaked). If there are a lot of files this may > take some time, rebooting will not speed it up. > > > On Oct 14, 2022, at 10:00, Cloete, F. (Francois) via lustre-discuss > <lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org%3cmailto:lustre-discuss@lists.lustre.org>>> > wrote: > > Hi Community, > Is anyone aware of a recycle bin parameter for Lustre? > > Just deleted a whole lot of files but for some reason the space is not > getting cleared. > > Server rebooted, file-system un-mounted etc. > > Thanks > > ******************** > Nedbank Limited group of companies (Nedbank) disclaimer and confidentiality > notice: > This email, including any attachments (email), contains information that is > confidential and is meant only for intended recipients. You may not share or > copy the email or any part of it, unless the sender has specifically allowed > you to do so. If you are not an intended recipient, please delete the email > permanently and let Nedbank know that you have deleted it by replying to the > sender or calling the Nedbank Contact Centre on 0860 555 111. > This email is not confirmation of a transaction or a Nedbank statement and is > not offering or inviting anyone to take up any financial products or > services, unless the content specifically indicates that it does so. Nedbank > will not be liable for any errors or omissions in this email. The views and > opinions are those of the author and not necessarily those of Nedbank. > The names of the Nedbank Board of Directors and Company Secretary are > available here: http://www.nedbank.co.za/terms/DirectorsNedbank.htm > <http://www.nedbank.co.za/terms/DirectorsNedbank.htm > <http://www.nedbank.co.za/terms/DirectorsNedbank.htm%20> >. Nedbank Ltd Reg > No 1951/000009/06. > ******************** > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org%3cmailto:lustre-discuss@lists.lustre.org> > > > https://eur01.safelinks.protection.outlook.com/?url=http://lists > .lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=05%7C0 > 1%7CFrancoisClo%40Nedbank.co.za%7Caccd70e00cb84df3418608dab017807f%7C0 > b1d23d810d140938cb7fd0bb32f81e1%7C0%7C0%7C638015912276367615%7CUnknown > %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ > XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f4pQ6fAnpWfLFoihgUUGBNXH65%2B9Yc > Wgotw5E7UuUgU%3D&reserved=0 > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Whamcloud > > > > > > > _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=05%7C01%7CFrancoisClo%40Nedbank.co.za%7Ccd127790e4714b6ffa0908dab0714a6d%7C0b1d23d810d140938cb7fd0bb32f81e1%7C0%7C0%7C638016297920336453%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rHCHfCCnDwn38lq8HE1QeF1a6yaLttJ%2BFnb1B08iEZ0%3D&reserved=0>
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
