Hey everyone, I had an OST crash (actually it made the entire OSS unresponsive to the point where I had to shoot it). There were messages in /var/log/messages complaining about slow journal performance (we have separate OSTs and journal disks).
Dec 28 20:29:02 oss-0-0 kernel: LustreError: 12638:0:(filter_io_26.c:742:filter_commitrw_write()) scratch-OST000e: slow direct_io 85s Dec 28 20:29:02 oss-0-0 kernel: LustreError: 12638:0:(filter_io_26.c:742:filter_commitrw_write()) Skipped 58 previous similar messages Dec 28 21:50:13 oss-0-0 kernel: LustreError: 16499:0:(lustre_fsfilt.h:320:fsfilt_commit_wait()) scratch-OST000e: slow journal start 51s Dec 28 21:50:13 oss-0-0 kernel: LustreError: 16499:0:(lustre_fsfilt.h:320:fsfilt_commit_wait()) Skipped 66 previous similar messages Lustre and e2fsprogs versions: [r...@oss-0-0 ~]# rpm -q kernel-lustre kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1 [r...@oss-0-0 ~]# rpm -q e2fsprogs e2fsprogs-1.41.6.sun1-0redhat Then there's this interesting message: Dec 29 14:11:32 oss-0-0 kernel: LDISKFS-fs error (device sdz): ldiskfs_lookup: unlinked inode 5384166 in dir #145170469 Dec 29 14:11:32 oss-0-0 kernel: Remounting filesystem read-only Followed by a whole lot of these: Dec 29 14:11:32 oss-0-0 kernel: LustreError: 18559:0:(fsfilt-ldiskfs.c:336:fsfilt_ldiskfs_start()) error starting handle for op 8 (106 credits): rc -30 Dec 29 14:11:33 oss-0-0 kernel: LustreError: 12648:0:(fsfilt-ldiskfs.c:336:fsfilt_ldiskfs_start()) error starting handle for op 8 (106 credits): rc -30 Dec 29 14:11:34 oss-0-0 kernel: LustreError: 16430:0:(fsfilt-ldiskfs.c:336:fsfilt_ldiskfs_start()) error starting handle for op 8 (106 credits): rc -30 Dec 29 14:11:35 oss-0-0 kernel: LustreError: 16446:0:(fsfilt-ldiskfs.c:470:fsfilt_ldiskfs_brw_start()) can't get handle for 125 credits: rc = -30 Dec 29 14:11:35 oss-0-0 kernel: LustreError: 16446:0:(filter_io_26.c:684:filter_commitrw_write()) error starting transaction: rc = -30 Dec 29 14:11:35 oss-0-0 kernel: LustreError: 12666:0:(filter_io_26.c:684:filter_commitrw_write()) error starting transaction: rc = -30 Dec 29 14:11:36 oss-0-0 kernel: LustreError: 16417:0:(fsfilt-ldiskfs.c:470:fsfilt_ldiskfs_brw_start()) can't get handle for 471 credits: rc = -30 Dec 29 14:11:36 oss-0-0 kernel: LustreError: 16417:0:(fsfilt-ldiskfs.c:470:fsfilt_ldiskfs_brw_start()) Skipped 1 previous similar message Dec 29 14:11:36 oss-0-0 kernel: LustreError: 16417:0:(filter_io_26.c:684:filter_commitrw_write()) error starting transaction: rc = -30 Whenever I try to mount the ost (known as /dev/dsk/ost24) I get the following messages: Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs error (device sdz): ldiskfs_check_descriptors: Checksum for group 16303 failed (64812!=44) Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs: group descriptors corrupted! Dec 29 19:25:35 oss-0-0 kernel: LustreError: 15067:0:(obd_mount.c:1278:server_kernel_mount()) premount /dev/dsk/ost24:0x0 ldiskfs failed: -22, ldiskfs2 failed: -19. Is the ldi skfs module available? Dec 29 19:25:35 oss-0-0 kernel: LustreError: 15067:0:(obd_mount.c:1592:server_fill_super()) Unable to mount device /dev/dsk/ost24: -22 Dec 29 19:25:35 oss-0-0 kernel: LustreError: 15067:0:(obd_mount.c:1997:lustre_fill_super()) Unable to mount (-22) Dec 29 19:25:35 oss-0-0 Filesystem[15020]: ERROR: Couldn't mount filesystem /dev/dsk/ost24 on /mnt/scratch/ost24 Dec 29 19:25:35 oss-0-0 Filesystem[15009]: ERROR: Generic error So it looks like the "group descriptors" are corrupted. I'm not sure what those are but e2fsck -n sure enough complains about them. So I tried running it for real. I ran e2fsck -j /dev/$JOURNAL -v -fy -C 0 /dev/$DEVICE. The first time I ran to what looked like completion. It printed a summary and all but then didn't exit. I sent it a kill but that didn't stop it. So I let it run and went back to sleep for 3 hours. When I woke up the process was gone but I still get the same error messages. I found this discussion http://lists.lustre.org/pipermail/lustre-discuss/2009-March/009885.html and tried the tune2fs command followed by the e2fsck but it hasn't exited yet (its a 2.7 TB OST) The LUN comes from a Sun STK 6140/CSM200 device which isn't reporting any warning, events, or errors. I deactivated the OST with lctl but it still shows up as active on the clients. Also lfs find /scratch -O scratch-OST000e_UUID HANGS! Are we screwed here? Is there a way to run lfs find with the OST disabled? Shouldn't that just be a metadata operation? Thanks Erik
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
