On May 31, 2022, at 13:01, William D. Colburn <wcolb...@nrao.edu<mailto:wcolb...@nrao.edu>> wrote:
We had a filesystem corruption back in February, and we've been trying to salvage things since then. I've spent the past month slowly draining the corrupt OST, and over the weekend it finally finished. An lfs find on the filesystem says that thhow ere are no files stored on that OST. The OST is 100% full, and if I mount it as an ldiskfs I can see a little over five millions files in O/*/*. How did you drain the OST? Was the OST totally deactivated, or "max_create_count=0"? If it was deactivated, then this will prevent OST objects from being destroyed when the MDT inode is deleted. Most of them have numbers as names, and some of them are named LAST_ID. This is normal. The number is the object ID. You can check the OST objects with ll_decode_filter_fid (when mounted as type ldiskfs) to report the parent MDT FID that the object belongs/belonged to. Then "lfs fid2path" can be used to check if the file still exists and/or if the OST object is still part of the layout (which it should not be). All of the numbered files seem to be user data, with owners, and real data in them (based on ls and the find command) I would like to clean out this OST and readd it to lustre, but I'm unsure of how to best approach this. I see several options: OPTION ONE: run lfsck against the entire filesystem with the full and previously corrupt OST mounted. This is unnecessary, since these objects are not used. With the "--orphan" option it will link the OST objects into $mount/.lustre/lost+found where they could be deleted, essentially the same as #4 below, but it is easier to just delete the objects directly if you know they are not needed. OPTION TWO: run lfsck against only the corrupt OST in the hopes that cleans up all of the orphans on that OST. This won't help, since the OST LFSCK would not attach the object into the visible namespace, so it won't really change the situation. OPTION THREE: mounted as ldiskfs remove O/*/[1234567890]*[1234567890] and then remount the file system. This would be one option. Note that with DNE the OST object names will be in hex, so the above regexp would not catch all objects. OPTION FOUR: newfs the bad OST and readd it losing the old index. This would also work, if you use the "--replace" option when formatting. We tried option one once before, and it killed cluster jobs because it made files unreadable while they were in use. Option two might avoid that since it would not be affecting existing files. Option three sounds like it will work based on my limited knowledge of how lustre works, and would probably be the most expedient method. Option four is annoying because it leaves a hole in the lustre that is upsetting to our OCD tendencies. Any and all advice is appreciated here. Thank you. --Schlake Sysadmin IV, NRAO Work: 575-835-7281 (BACK IN THE OFFICE!) Cell: 575-517-5668 (out of work hours) _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Whamcloud
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org