Hi, During the service break e2fsck revealed that one user had a corrupted directory structure on MDT. I've tried to run e2fsck -f multiple times but it cannot fix this problem. Mounting MDT as a lustre OI-scrub will start and after some time MDS will hit LBUG. I think next step could be just remove problematic directory with debugfs clri <inode> but do you have any better suggestions? (fs was formatted with lustre 2.1.3 tools, upgraded to 2.5.x later)
e2fsck 1.42.13.wc4 (28-Nov-2015) MMP interval is 10 seconds and total wait time is 42 seconds. Please wait... wrk-MDT0000: recovering journal Pass 1: Checking inodes, blocks, and sizes Inodes that were part of a corrupted orphan linked list found. Fix<y>? yes Inode 397458656 was part of the orphaned inode list. FIXED. Inode 397458656, i_blocks is 3118989520, should be 0. Fix<y>? yes Inode 397458688 has illegal block(s). Clear<y>? yes Illegal block #2 (2730544606) in inode 397458688. CLEARED. Illegal block #4 (2631036428) in inode 397458688. CLEARED. Illegal block #5 (1163698720) in inode 397458688. CLEARED. Illegal block #6 (4018192778) in inode 397458688. CLEARED. Illegal block #8 (2730544607) in inode 397458688. CLEARED. Illegal indirect block (3417817744) in inode 397458688. CLEARED. Inode 397458688, i_size is 1414676816, should be 49152. Fix<y>? yes Inode 397458688, i_blocks is 1, should be 48. Fix<y>? yes Running additional passes to resolve blocks claimed by more than one inode... Pass 1B: Rescanning for multiply-claimed blocks Multiply-claimed block(s) in inode 397458688: 34 1 1 128 128 Pass 1C: Scanning directories for inodes with multiply-claimed blocks Pass 1D: Reconciling multiply-claimed blocks (There are 1 inodes containing multiply-claimed blocks.) File ... (inode #397458688, mod time Mon Jun 7 11:03:47 1971) has 5 multiply-claimed block(s), shared with 1 file(s): <filesystem metadata> Clone multiply-claimed blocks<y>? yes Pass 2: Checking directory structure Directory inode 397458688, block #0, offset 0: directory corrupted Salvage<y>? yes Missing '.' in directory inode 397458688. Fix<y>? yes Inode 397458688 is badly corrupt (badness value = 10). Clear<y>? yes Missing '..' in directory inode 397458688. Fix<y>? yes Setting filetype for entry '..' in ??? (397458688) to 2. Pass 3: Checking directory connectivity Pass 3A: Optimizing directories Failed to optimize directory ??? (397458688): EXT2 directory corrupted Pass 4: Checking reference counts Inode 2 ref count is 12, should be 13. Fix<y>? yes Inode 397458656 (...) has invalid mode (0151571). Clear<y>? yes Pass 5: Checking group summary information Free blocks count wrong for group #0 (2100, counted=2095). Fix<y>? yes Free blocks count wrong for group #4878 (15003, counted=15002). Fix<y>? yes Free blocks count wrong (441277870, counted=440655796). Fix<y>? yes Free inodes count wrong for group #12129 (149, counted=147). Fix<y>? yes Directories count wrong for group #12129 (2655, counted=2656). Fix<y>? yes Free inodes count wrong (530993943, counted=530993800). Fix<y>? yes [QUOTA WARNING] Usage inconsistent for ID 0:actual (16773439488, 22078) != expected (14225448960, 21937) [QUOTA WARNING] Missing quota entry ID 8265 [QUOTA WARNING] Missing quota entry ID 59094 Update quota info for quota type 0<y>? yes [QUOTA WARNING] Usage inconsistent for ID 0:actual (16773435392, 30288) != expected (14225444864, 30147) [QUOTA WARNING] Missing quota entry ID 16008 [QUOTA WARNING] Missing quota entry ID 52735 Update quota info for quota type 1<y>? yes wrk-MDT0000: ***** FILE SYSTEM WAS MODIFIED ***** wrk-MDT0000: 811183480/1342177280 files (0.1% non-contiguous), 230432844/671088640 blocks debugfs 1.42.13.wc4 (28-Nov-2015) /dev/mapper/lustre-MDT_wrk: catastrophic mode - not reading inode or group bitmaps Inode: 397458688 Type: directory Mode: 0105 Flags: 0x0 Generation: 0 Version: 0x00000000 User: 8265 Group: 52735 Size: 49152 File ACL: 0 Directory ACL: 0 Links: 0 Blockcount: 48 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x0000005c -- Thu Jan 1 02:01:32 1970 atime: 0x00010000 -- Thu Jan 1 20:12:16 1970 mtime: 0x02b0ae73 -- Mon Jun 7 11:03:47 1971 dtime: 0x56bb5e7a -- Wed Feb 10 17:59:54 2016 Size of extra inode fields: 0 BLOCKS: (0):13820, (3):13821, (7):79926036, (9):13822, (10):13834, (11):13856 TOTAL: 6 _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
