I just wanted to confirm that I saw just this exact same issue today on a stock RHEL 4 Update 5. Running gfs_tool reclaim made my file system "work" again.
# rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort ccs-1.0.10-0 ccs-devel-1.0.10-0 cman-1.0.17-0 cman-devel-1.0.17-0 cman-kernel-xenU-2.6.9-50.2 cman-kernheaders-2.6.9-50.2 GFS-6.1.14-0 GFS-kernel-xenU-2.6.9-72.2 magma-1.0.7-1 magma-plugins-1.0.12-0 # uname -rm 2.6.9-55.ELxenU x86_64 2008/7/15 Reiner Rottmann <[EMAIL PROTECTED]> > Hello everyone, > > I've experienced strange behavior on a 20 GB GFS formatted volume (although > same behaviour applies to smaller and larger sizes) when reaching the max > available disk space by writing lots of 256 byte files in a nested > directory > structure (~15k files in one dir). > > The expected behaviour would be that all free data blocks are transformed > to > inodes and metadata as required but although there are still plenty > datablocks free, new 256 byte files cannot be created due to "No space left > on device". > > After that, when creating sequential files via touch, it is expected that > they > are created till all data blocks are transformed in inodes representing the > files. When all data blocks are used, "No space left on device" is > expected. > But in this strange scenario, files are created at random!? > > Also when executing gfs_tool reclaim, new files are createable again. But > gfs_tool reclaim only should increase the number of already available free > data blocks by cleaning unused metadata blocks. > > In my understanding, it should not be necessary to reclaim blocks, if there > are still free data blocks left. > > Has anyone an explanation for this? > > Best regards, > > Reiner Rottmann > > > --%<--------------------------------------------------------------------------- > (Filesystem filled with 256 byte files.) > > # for i in $(seq 1 1000); do touch waste.$i; done > touch: cannot touch `waste.3': No space left on device > touch: cannot touch `waste.6': No space left on device > touch: cannot touch `waste.12': No space left on device > touch: cannot touch `waste.13': No space left on device > touch: cannot touch `waste.15': No space left on device > touch: cannot touch `waste.16': No space left on device > touch: cannot touch `waste.20': No space left on device > touch: cannot touch `waste.25': No space left on device > touch: cannot touch `waste.28': No space left on device > touch: cannot touch `waste.29': No space left on device > touch: cannot touch `waste.32': No space left on device > touch: cannot touch `waste.37': No space left on device > touch: cannot touch `waste.38': No space left on device > touch: cannot touch `waste.39': No space left on device > touch: cannot touch `waste.48': No space left on device > touch: cannot touch `waste.55': No space left on device > touch: cannot touch `waste.56': No space left on device > touch: cannot touch `waste.59': No space left on device > touch: cannot touch `waste.60': No space left on device > touch: cannot touch `waste.63': No space left on device > ^C > > # for i in $(seq 1 1000); do touch waste2.$i; done > touch: cannot touch `waste2.1': No space left on device > touch: cannot touch `waste2.8': No space left on device > touch: cannot touch `waste2.10': No space left on device > touch: cannot touch `waste2.11': No space left on device > touch: cannot touch `waste2.12': No space left on device > touch: cannot touch `waste2.14': No space left on device > touch: cannot touch `waste2.17': No space left on device > touch: cannot touch `waste2.19': No space left on device > touch: cannot touch `waste2.21': No space left on device > touch: cannot touch `waste2.24': No space left on device > touch: cannot touch `waste2.28': No space left on device > touch: cannot touch `waste2.31': No space left on device > touch: cannot touch `waste2.32': No space left on device > touch: cannot touch `waste2.33': No space left on device > touch: cannot touch `waste2.40': No space left on device > touch: cannot touch `waste2.43': No space left on device > touch: cannot touch `waste2.44': No space left on device > touch: cannot touch `waste2.49': No space left on device > touch: cannot touch `waste2.54': No space left on device > touch: cannot touch `waste2.55': No space left on device > touch: cannot touch `waste2.57': No space left on device > touch: cannot touch `waste2.58': No space left on device > touch: cannot touch `waste2.61': No space left on device > ^C > > # gfs_tool df . > /mnt/gfstest: > SB lock proto = "lock_dlm" > SB lock table = "axqa01:gfstest" > SB ondisk format = 1309 > SB multihost format = 1401 > Block size = 1024 > Journals = 3 > Resource Groups = 78 > Mounted lock proto = "lock_dlm" > Mounted lock table = "axqa01:gfstest" > Mounted host data = "" > Journal number = 0 > Lock module flags = > Local flocks = FALSE > Local caching = FALSE > Oopses OK = FALSE > > Type Total Used Free use% > ------------------------------------------------------------------------ > inodes 18343309 18343309 0 100% > metadata 1690156 1687524 2632 100% > data 43931 0 43931 0% > > # rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort > GFS-6.1.15-1 > GFS-kernel-2.6.9-60.9 > GFS-kernel-2.6.9-75.11 > GFS-kernel-smp-2.6.9-60.9 > GFS-kernel-smp-2.6.9-75.11 > ccs-1.0.11-1 > cman-1.0.17-0.el4_6.3 > cman-kernel-smp-2.6.9-45.15 > cman-kernel-smp-2.6.9-53.8 > magma-1.0.8-1 > magma-devel-1.0.8-1 > magma-plugins-1.0.12-0 > > # cat /etc/redhat-release > Red Hat Enterprise Linux AS release 4 (Nahant Update 6) > > # uname -a > Linux realserver10 2.6.9-67.0.4.ELsmp #1 SMP Fri Jan 18 05:00:00 EST 2008 > x86_64 x86_64 x86_64 GNU/Linux > > --%<--------------------------------------------------------------------------- > > -- > Gruss / Regards, > > Dipl.-Ing. (FH) Reiner Rottmann > > Phone: +49-89 452 3538-12 > > http://www.atix.de/ > http://open-sharedroot.org/ > > PGP Key ID: 0xCA67C5A6 > PGP Key Fingerprint = BF59FF006360B6E8D48F26B10D9F5A84CA67C5A6 > > ** > ATIX Informationstechnologie und Consulting AG > Einsteinstr. 10 > 85716 Unterschleissheim > Deutschland/Germany > > Phone: +49-89 452 3538-0 > Fax: +49-89 990 1766-0 > > Registergericht: Amtsgericht Muenchen > Registernummer: HRB 168930 > USt.-Id.: DE209485962 > > Vorstand: > Marc Grimme, Mark Hlawatschek, Thomas Merz (Vors.) > > Vorsitzender des Aufsichtsrats: > Dr. Martin Buss > > -- > Linux-cluster mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Dave Costakos mailto:[EMAIL PROTECTED]
-- Linux-cluster mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-cluster
