Hi Again, I guess I spoke too soon. Looks like the MDS is full, even though it still has plenty of space, it's reporting that it's reached it's inode limit. Not sure how this is fixed. I am also having a hard time finding information about this (I wish PDF text was searchable).
# lfs df -h UUID bytes Used Available Use% Mounted on clients-MDT0000_UUID 35.0G 9.4G 23.6G 26% /data[MDT:0] clients-OST0000_UUID 2.0T 319.5G 1.6T 15% /data[OST:0] filesystem summary: 2.0T 319.5G 1.6T 15% /data # lfs df -i UUID Inodes IUsed IFree IUse% Mounted on clients-MDT0000_UUID 10485760 10409292 76468 99% /data[MDT:0] clients-OST0000_UUID 134217728 8082612 126135116 6% /data[OST:0] filesystem summary: 10485760 10409292 76468 99% /data I tried to grow the MDT +50G with the following: # lvextend -L +50G /dev/vg-localdisk/mdt Extending logical volume mdt to 90.00 GB Logical volume mdt successfully resized # resize2fs -p /dev/vg-localdisk/mdt resize2fs 1.40.11.sun1 (17-June-2008) resize2fs: Filesystem has unsupported read-only feature(s) :- uninit_groups Didn't seem to work. So I've got a 90G LVM slice allocated to the MDT which is only using 35G of that (can't get it to grow to use the rest). Of that 35G, 24G is still free but my inode limit is at 99% and I'm not sure how to extend that. Thanks for any help. -Nick On Tue, 2009-09-29 at 11:47 +0200, Nick Jennings wrote: > Hi Everyone, > > RHEL5.3 - Lustre 1.6.7.2 - filesystem mounted as /data. > > I've got another odd problem this morning. Suddenly the file system was > reporting "no space left on device" failing every write attempt. A df > showed still 1.7TB free. > > I first tried unmounting and remounting the /data filesystem. However > on remount I got the error: > > # mount /data > mount.lustre: mount id...@tcp1:/clients at /data failed: Cannot send > after transport endpoint shutdown > > I then realized top was reporting one of the Lustre threads on the MDT > at 100% CPU. So I unmounted and remounted the MDT partition on that > server, and could then remount the /data filesystem on the client node. > > Now everything seems to be back to normal operation. Any idea what was > going on there? I've got absolutely nothing from any of the log files. > > Thanks for any help, > -Nick > > > - > Nick Jennings > Technical Director > Creative Motion Design > www.creativemotiondesign.com >
signature.asc
Description: This is a digitally signed message part
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
