And just to add one more item to the list: If you are running ZFS as the backend storage and have enabled compression, you need to use an additional flag to du:
du -sh <dir> Will give you what is stored on the disk du --apparent-size <dir> will tell you what the actual size of the data is. I doubt this is what you are running into, but it's useful knowledge to share. -Marc ---- D. Marc Stearman Lustre Operations Lead [email protected] 925.423.9670 On May 21, 2015, at 9:36 AM, Martin Hecht <[email protected]> wrote: > Hi, > > a few more things which may play a role: > > - as you are suspecting, the difference of used blocks vs. used bytes > might be the reason, especially if there are many very small files, but > there are more possible causes: > > - some tools use 2^10 bytes and some others use 1000 bytes as kb which > might explain small discrpancies. du and ls are examples for different > output. However, this can not explain the whole difference. > > - your find looks for regular files only (type f), but directories and > symbolic links consume a few kb as well. If there are many symbolic > links, this would be the explanation, I think. > > - there are also cases in which quota can get out of sync (I don't > remember the cause, but I have already seen warnings about this in the > syslog of lustre servers). e2fsck on the ldiskfs level is supposed to > fix this issue, but I also had cases in which I had to turn off quota by > means of tune2fs on the ldisk level and turn it on again in order to > trigger something like a background quotacheck in lustre 2. In lustre > 1.8 there used to be a tool "quotacheck" > > - preallocated stripes on many ost's might be an issue as well, although > I don't see the discrepancy described on our file systems. > > - there might also be orphane objects on the disk, i.e. stripes which > are not referenced anymore on the lustre level, but which still consume > disk space (not sure if these may affect quota). An online lfsck is > supposed to clean them up in lustre 2. In lustre 1.8 one had to run > several e2fsck on the ldisk level and build databases to run an lfsck, > but that's not supported anymore in lustre 2. > > best regards, > Martin > > On 05/20/2015 10:50 AM, Phill Harvey-Smith wrote: >> Hi all, >> >> One of my users is reporting a massive size difference between the >> figures reported by du and quota. >> >> doing a du -hs on his directory reports : >> du -hs . >> 529G . >> >> doing a lfs quota -u username /storage reports >> Filesystem kbytes quota limit grace files quota limit >> grace >> /storage 621775192 640000000 640010000 - 601284 1000000 >> 1100000 - >> >> Though this user does have a lot of files : >> >> find . -type f | wc -l >> 581519 >> >> So I suspect that it is the typical thing that quota is reporting used >> blocks whilst du is reporting used bytes, which can of course be >> wildly different due to filesystem overhead and wasted unused space at >> the end of files where a block is allocated but only partially used. >> >> Is this likely to be the case ? >> >> I'm also not entirely sure what versions of lustre the client machines >> and MDS / OSS servers are running, as I didn't initially set the >> system up. >> >> Cheers. >> >> Phill. >> >> _______________________________________________ >> lustre-discuss mailing list >> [email protected] >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > > > _______________________________________________ > lustre-discuss mailing list > [email protected] > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
