Never mind I figured it out. First you have to mount -t ldiskfs /dev/OSTDEV /mnt/OST Then you can run ll_recover_lost_found_objs.
Erik On Sat, Jan 2, 2010 at 12:33 PM, Erik Froese <[email protected]> wrote: > Thanks Andreas, > > After I e2fsck'd the OST again (and once more to make sure there weren't > any errors) the OST mounted cleanly. > I'm not able to run the ll_recover_lost_found_objs. > > [r...@oss-0-0 ~]# ll_recover_lost_found_objs -d > /mnt/scratch/ost24/lost+found | tee /tmp/ll_recover_lost_found_obj.ost24 > error: creating objects directory /mnt/scratch/ost24/O: Not a directory > "lost+found" directory path: /mnt/scratch/ost24/lost+found > > The lustre FS is mounted at /scratch on the clients and I don't see a > lost+found directory there either. > > Erik > > On Sat, Jan 2, 2010 at 1:13 AM, Andreas Dilger <[email protected]> wrote: > >> On 2010-01-01, at 17:20, Erik Froese wrote: >> >>> On Thu, Dec 31, 2009 at 4:52 PM, Andreas Dilger <[email protected]> wrote: >>> >>>> These are usually a sign that the back-end storage is overloaded, or >>>> somehow performing very slowly. Maybe there was a RAID rebuild going on? >>>> >>> >>> I don't see any hw failures or rebuild messages on the RAID. I do see >>> periodic media scans going on. >>> >> >> There definitely was corruption of the directories on the OST, consistent >> with what the kernel originally reported, but it looks like it was resolved >> with a minimum of further problems. You probably want to run the >> "ll_recover_lost_found_objs" on this OST to restore the objects from >> lost+found back to their correct locations, though that isn't strictly >> necessary to bring the OST back online - it just avoids losing access to the >> objects that were moved to lost+found due to the directory corruption. >> >> Lustre and e2fsprogs versions: >>> >>> [r...@oss-0-0 ~]# rpm -q kernel-lustre >>> kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1 >>> [r...@oss-0-0 ~]# rpm -q e2fsprogs >>> e2fsprogs-1.41.6.sun1-0redhat >>> >>> >>> Then there's this interesting message: >>> Dec 29 14:11:32 oss-0-0 kernel: LDISKFS-fs error (device sdz): >>> ldiskfs_lookup: unlinked inode 5384166 in dir #145170469 >>> Dec 29 14:11:32 oss-0-0 kernel: Remounting filesystem read-only >>> >>> This means the ldiskfs code found some corruption on disk, and remounted >>> the filesystem read-only to avoid further corruptions on disk. >>> >>> >>> Whenever I try to mount the ost (known as /dev/dsk/ost24) I get the >>> following messages: >>> Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs error (device sdz): >>> ldiskfs_check_descriptors: Checksum for group 16303 failed (64812!=44) >>> Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs: group descriptors corrupted! >>> >>> So it looks like the "group descriptors" are corrupted. I'm not sure what >>> those are but e2fsck -n sure enough complains about them. So I tried running >>> it for real. >>> >>> I ran e2fsck -j /dev/$JOURNAL -v -fy -C 0 /dev/$DEVICE. >>> >>> The first time I ran to what looked like completion. It printed a summary >>> and all but then didn't exit. I sent it a kill but that didn't stop it. So I >>> let it run and went back to sleep for 3 hours. When I woke up the process >>> was gone but I still get the same error messages. >>> >>> Having a log of the e2fsck errors would be helpful. >>> >>> I don't have the log for the first fsck. First it logged a bunch of >>> messages about the group descriptors and that it was repairing them. Then >>> messages about inodes. I'm attaching the logs from the second e2fsck. >>> >>> >>> >>> >>> >>> I found this discussion >>> http://lists.lustre.org/pipermail/lustre-discuss/2009-March/009885.html >>> and tried the tune2fs command followed by the e2fsck but it hasn't exited >>> yet (its a 2.7 TB OST) >>> >>> It might take an hour or two, depending on how fast your storage is. >>> >>> >>> The LUN comes from a Sun STK 6140/CSM200 device which isn't reporting any >>> warning, events, or errors. >>> >>> I deactivated the OST with lctl but it still shows up as active on the >>> clients. Also lfs find /scratch -O scratch-OST000e_UUID HANGS! >>> >>> You also need to deactivate it on the clients, at which point they will >>> get an IO error when accessing files on that OST. >>> >>> I didn't know that the you had to disable it on the clients as well. >>> Thanks >>> >>> >>> Are we screwed here? Is there a way to run lfs find with the OST >>> disabled? Shouldn't that just be a metadata operation? >>> >>> >>> The size of a file is stored on the OSTs, so it depends on what you are >>> trying to do. "lfs getstripe" can be run with a deactivated OST. >>> >>> >>> Cheers, Andreas >>> -- >>> Andreas Dilger >>> Sr. Staff Engineer, Lustre Group >>> Sun Microsystems of Canada, Inc. >>> >>> >>> <e2fsck-fy.sdz.out.post-mmp> >>> >> >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Sr. Staff Engineer, Lustre Group >> Sun Microsystems of Canada, Inc. >> >> >
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
