Sorry about the delay on this. I managed to get some downtime on the system and the "debugfs.ocfs2 -f" command did the trick. I have taken a snapshot of the lun while it was in the bad state so if I get some time I will try and investigate it further.
Srinivas, Is there anything you would like me to run on this snapshot to get some diagnostics for you or will i need to have those traces running when the issue initially occures ? In its current state anything I delete from the snapshot causes orphans. Would that be enough to get some traces out of it? Tim Hughes www.tradefair.com Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London | W11 4AN The information in this e-mail and any attachment is confidential and is intended only for the named recipient(s). The e-mail may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not a named recipient please notify the sender immediately and delete any copies of this message. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any view or opinions presented are solely those of the author and do not necessarily represent those of the company. ----- Original Message ----- From: "Srinivas Eeda" <srinivas.e...@oracle.com> To: "Tim Hughes" <tim.hug...@tradefair.com> Cc: ocfs2-users@oss.oracle.com Sent: Friday, 27 February, 2009 10:21:50 PM GMT +00:00 GMT Britain, Ireland, Portugal Subject: Re: [Ocfs2-users] how do I remove ocfs2 orphaned files that are taking up diskspace ? please run fsck.ocfs2 -f to clean the orphans. If you can reproduce it at will, can you please enable tracing debugfs.ocfs2 -l JOURNAL, SUPER, INODE, NAMEI, DLM_GLUE ENTRY EXIT allow and send us the messages files from all nodes. thanks, --Srini Tim Hughes wrote: > I am looking for a little help with some orphaned files that are taking up > diskspace. > > I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files from a three > node ocfs2 cluster. The files appeared removed from the filesystem but the > results of a `df -h /var/lib/mysql` showed that no disk space has been > cleared. A `du -sh /var/lib/mysql` on the other hand says that ~ 30GB was > removed. > > <snip> > [r...@host1 mysql]# du -sh /var/lib/mysql > 50G /var/lib/mysql > [r...@host1 mysql]# df -h /var/lib/mysql/ > Filesystem Size Used Avail Use% Mounted on > /dev/xvdb1 100G 83G 18G 83% /var/lib/mysql > [r...@host1 mysql]# > <snip> > > First thoughts were that mysql was holding the file descriptors open but > `lsof` showed nothing. After trying a few other things we discovered that the > files were orphans of ocfs2. > > <snip> > [root@ host1 ~]# debugfs.ocfs2 -R "ls -l //orphan_dir:0001" /dev/xvdb1 > 13 drwxr-xr-x 2 0 0 4096 23-Feb-2009 21:54 . > 6 drwxr-xr-x 18 0 0 4096 20-Aug-2008 15:54 .. > 1064634 -rw-rw---- 0 100 101 1024 5-Sep-2008 14:14 0000000000103eba > 1064635 -rw-rw---- 0 100 101 0 5-Sep-2008 14:14 0000000000103ebb > 1064633 -rw-rw---- 0 100 101 8554 5-Sep-2008 14:14 0000000000103eb9 > 1064637 -rw-rw---- 0 100 101 114688 1-Oct-2008 12:33 0000000000103ebd > 1064636 -rw-rw---- 0 100 101 8844 1-Oct-2008 11:37 0000000000103ebc > 1193689 -rw-rw---- 0 100 101 1073742258 21-Jan-2009 15:18 00000000001236d9 > 1193690 -rw-rw---- 0 100 101 1073741942 21-Jan-2009 20:10 00000000001236da > 1193691 -rw-rw---- 0 100 101 1073743434 22-Jan-2009 14:29 00000000001236db > 1193692 -rw-rw---- 0 100 101 1073742303 22-Jan-2009 18:37 00000000001236dc > 1193693 -rw-rw---- 0 100 101 1073741875 23-Jan-2009 12:59 00000000001236dd > 1193694 -rw-rw---- 0 100 101 1073741973 23-Jan-2009 18:36 00000000001236de > 1193695 -rw-rw---- 0 100 101 1073742198 26-Jan-2009 14:05 00000000001236df > 1193696 -rw-rw---- 0 100 101 1073742221 26-Jan-2009 20:18 00000000001236e0 > 1193697 -rw-rw---- 0 100 101 1073742068 27-Jan-2009 14:46 00000000001236e1 > 1193698 -rw-rw---- 0 100 101 1005225645 28-Jan-2009 00:06 00000000001236e2 > <snip> > > We have shutdown mysql and tried unmounting and mounting the ocfs2 file > system from the node where we deleted the files. This made no difference so > we decided to take down the mysql cluster and unmounting the ocfs2 filesystem > from all nodes just incase something was for some reason holding these files > open that we couldn't find. This didn't clear the space and the orphans were > still there. > > Next we unmounted it all again and ran `fsck.ocfs2 /dev/xvdb1` on it with the > following results which indicated there was nothing wrong. > > <snip> > Checking OCFS2 filesystem in /dev/xvdb1: > label: db-store01 > uuid: 74 c1 14 6c af ee 4e 29 84 e4 c1 7a a8 cc 96 eb > number of blocks: 26214047 > bytes per block: 4096 > number of clusters: 26214047 > bytes per cluster: 4096 > max slots: 16 > > /dev/xvdb1 is clean. It will be checked after 20 additional mounts. > <snip> > > We are running : > ocfs2-tools-1.4.1-1.el5 > ocfs2-2.6.18-92.1.10.el5xen-1.4.1-1.el5 > > with kernel 2.6.18-92.1.10.el5xen on RHEL5 > > > How do we reclaim this space ? > > > > Tim Hughes > > www.tradefair.com > > Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London | W11 4AN > > The information in this e-mail and any attachment is confidential and is > intended only for the named recipient(s). The e-mail may not be disclosed or > used by any person other than the addressee, nor may it be copied in any way. > If you are not a named recipient please notify the sender immediately and > delete any copies of this message. Any unauthorized copying, disclosure or > distribution of the material in this e-mail is strictly forbidden. Any view > or opinions presented are solely those of the author and do not necessarily > represent those of the company. > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users@oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users >
_______________________________________________ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
