Re: Can somebody explain this? du & ls showing different sizes
On Fri, 2009-02-20 at 13:15 -0800, Leslie Satenstein wrote: > > > --- On Mon, 2/16/09, Patrick O'Callaghan > wrote: > From: Patrick O'Callaghan > Subject: Re: Can somebody explain this? du & ls showing > different sizes > To: fedora-list@redhat.com > Date: Monday, February 16, 2009, 8:45 AM > > On Mon, 2009-02-16 at 07:13 +, Vijay Gill wrote: > > 2009/2/16 Patrick O'Callaghan : > > > On Sun, 2009-02-15 at 23:45 +, Vijay Gill wrote: > > >> > That could be because you're using XFS. > > >> > > > >> > poc > > > [...] > > >> Thanks but how does that statement explain the behaviour I am > seeing? > > > > > > I'm > reaching here, but since XFS uses allocation strategies > different > > > from the more familiar ext3 system, I wondered if 'du' could > in some > > > circumstances report larger numbers than "normal". > > > > > > Apologies if I'm raving. > > > > > > poc > > > > > > -- > > > fedora-list mailing list > > > fedora-list@redhat.com > > > To unsubscribe: > https://www.redhat.com/mailman/listinfo/fedora-list > > > Guidelines: > http://fedoraproject.org/wiki/Communicate/MailingListGuidelines > > > > > > > You are spot on Pat. XFS allws you to pre-allocate contiguous area > of > > disk for a file when it is opened for writing data. This is > determined > > by allocsize parameter during mount and is used foe reducing > > fragmentation(fragmentation in Linux? yes siree, it does happen in > > Linux too). > > > > In my case I have provided 1M > which is far much less than 512M which > > XFS is pre-allocating. This is what caught my eye. > > It's nice to have one's intuition confirmed once in a while :-) > > poc > > Hi Poc > > Where is the overallocation parameter stored? > > Poc, I appreciate your expertise in the answers, but when you answer > something about Linux, > note that I have not your ideas of where to look for settings. > Therefore, I am sure readers would like a hint about the > directory/file in which this info > is stored. > > Thanks and have a good weekend. > > Leslie Actually I don't claim any specific expertise regarding XFS, but when you mess about with operating systems and filesystems for many many years, you develop a certain feeling for how things are likely to work (there's nothing new under the sun). As to where XFS stores the "overallocation parameter", I've no idea but the first place I'd look would be in the filesystem superblock, not in any file or directory. Or as a last resort you could always read the manual ... poc PS Please don't post in HTML. I can't edit it sensibly, which is why I just tacked a reply on the end. It's also contrary to the Guidelines of this list. -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
--- On Mon, 2/16/09, Patrick O'Callaghan wrote: From: Patrick O'Callaghan Subject: Re: Can somebody explain this? du & ls showing different sizes To: fedora-list@redhat.com Date: Monday, February 16, 2009, 8:45 AM On Mon, 2009-02-16 at 07:13 +, Vijay Gill wrote: > 2009/2/16 Patrick O'Callaghan : > > On Sun, 2009-02-15 at 23:45 +, Vijay Gill wrote: > >> > That could be because you're using XFS. > >> > > >> > poc > > [...] > >> Thanks but how does that statement explain the behaviour I am seeing? > > > > I'm reaching here, but since XFS uses allocation strategies different > > from the more familiar ext3 system, I wondered if 'du' could in some > > circumstances report larger numbers than "normal". > > > > Apologies if I'm raving. > > > > poc > > > > -- > > fedora-list mailing list > > fedora-list@redhat.com > > To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list > > Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines > > > > You are spot on Pat. XFS allws you to pre-allocate contiguous area of > disk for a file when it is opened for writing data. This is determined > by allocsize parameter during mount and is used foe reducing > fragmentation(fragmentation in Linux? yes siree, it does happen in > Linux too). > > In my case I have provided 1M which is far much less than 512M which > XFS is pre-allocating. This is what caught my eye. It's nice to have one's intuition confirmed once in a while :-) poc Hi Poc Where is the overallocation parameter stored? Poc, I appreciate your expertise in the answers, but when you answer something about Linux, note that I have not your ideas of where to look for settings. Therefore, I am sure readers would like a hint about the directory/file in which this info is stored. Thanks and have a good weekend. Leslie -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
On Monday 16 February 2009 07:13:23 Vijay Gill wrote: ... > In my case I have provided 1M which is far much less than 512M which > XFS is pre-allocating. This is what caught my eye. That might be 1M * 512 byte traditional disk sector size? > Cheers > Vijay -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
Latest kernel (kernel-2.6.27.12-170.2.5.fc10.i686): files under /var/log, /var/run and even MySQL database files get 512M allocated to them while they are open (which they do while system is running) even though their size is very small, leading to reporting of very less free space(a few megabytes) even though 3.1G should be reported. Previous Kernel (2.6.27.9-159.fc10.i686): There is no such behaviour and system reports correct free space. I will file bug in bugzilla this week-end after collecting more information instead of just some observations. I read the mail of that guy and you are right, he was talking of FC7. Vijay -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
On Wed, 2009-02-18 at 18:12 +, Vijay Gill wrote: > > What do you mean "the problem went away"? Do you mean that the 512M > > files created under the new kernel now showed only a few bytes > > allocated, or do you mean that *new* files created under the *old* > > kernel don't have the 512M allocation? > > > > poc > > > > PS Your mailer is including the entire trailer of list messages in the > > quoted material. This is dumb and most mailers don't do it. You should > > either edit it out or consider using a different mailer. > > > > By 'the problem went away' I meant that the files were no getting 512M > allocated for them even if they were a few bytes in size. Allocated *under which kernel*? You still haven't said. If this is a genuine bug you should report it to Bugzilla, but it's essential to be clear about what's happening. > The trouble it was causing is explained in following text. > > The files in question were in /var/log and /var/run. Most of them were > very small (ranging from 5-6 bytes to a few kilo bytes). But XFS > pre-allocated 512M to them. Once XFS pre-allocates space to a file, > the free space reported decreases by the amount that is pre-allocated > and not by the actual data written into the file. If these files are > closed then XFS would report free space left correctly. Since these > files were always open, my hard disk showed only a few megabytes free > even though it was supposed to have 3.1Gb free (as reported now). > > I have a suspicion that there is another linux user facing similar > problem (he also ran out of space in /var as he said). He might be > running the latest kernel. If I remember correctly, he was talking about installing (or reinstalling) an old system, something like FC7. > As far as my mailer is concerned I am using gmail (web interface). I > try to truncate the text as much as possible and keep text from only > previous mail so that the context is clear. It is as dumb as it is I > cannot help there. Well this time you did remove the trailing material, so that's fine. poc -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
> What do you mean "the problem went away"? Do you mean that the 512M > files created under the new kernel now showed only a few bytes > allocated, or do you mean that *new* files created under the *old* > kernel don't have the 512M allocation? > > poc > > PS Your mailer is including the entire trailer of list messages in the > quoted material. This is dumb and most mailers don't do it. You should > either edit it out or consider using a different mailer. > By 'the problem went away' I meant that the files were no getting 512M allocated for them even if they were a few bytes in size. The trouble it was causing is explained in following text. The files in question were in /var/log and /var/run. Most of them were very small (ranging from 5-6 bytes to a few kilo bytes). But XFS pre-allocated 512M to them. Once XFS pre-allocates space to a file, the free space reported decreases by the amount that is pre-allocated and not by the actual data written into the file. If these files are closed then XFS would report free space left correctly. Since these files were always open, my hard disk showed only a few megabytes free even though it was supposed to have 3.1Gb free (as reported now). I have a suspicion that there is another linux user facing similar problem (he also ran out of space in /var as he said). He might be running the latest kernel. As far as my mailer is concerned I am using gmail (web interface). I try to truncate the text as much as possible and keep text from only previous mail so that the context is clear. It is as dumb as it is I cannot help there. I hope I have explained things properly. Vijay -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
On Tue, 2009-02-17 at 23:29 +, Vijay Gill wrote: > 2009/2/17 Patrick O'Callaghan : > > On Tue, 2009-02-17 at 23:05 +, Vijay Gill wrote: > >> Back again with another observation. > >> > >> I found out that this issue happens only with > >> kernel-2.6.27.12-170.2.5.fc10.i686 (the latest one I have). I had a > >> previous kernel too (kernel-2.6.27.9-159.fc10.i686) and there is no > >> problem with that. I tried compare the change log of the two rpms and > >> there seemed nothing about changes to XFS. > > > > Does it make a difference which of the two kernels creates the file? [...] > Well most of the files which exihibit that issue were in /var/run or > /var/log. While investigating the latest kernel I had even cleared > /var/run still the files created were a few bytes in size but were > allocated 512M. I did nothing except rebooting to old kernel and > problem went away. What do you mean "the problem went away"? Do you mean that the 512M files created under the new kernel now showed only a few bytes allocated, or do you mean that *new* files created under the *old* kernel don't have the 512M allocation? poc PS Your mailer is including the entire trailer of list messages in the quoted material. This is dumb and most mailers don't do it. You should either edit it out or consider using a different mailer. -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
2009/2/17 Patrick O'Callaghan : > On Tue, 2009-02-17 at 23:05 +, Vijay Gill wrote: >> Back again with another observation. >> >> I found out that this issue happens only with >> kernel-2.6.27.12-170.2.5.fc10.i686 (the latest one I have). I had a >> previous kernel too (kernel-2.6.27.9-159.fc10.i686) and there is no >> problem with that. I tried compare the change log of the two rpms and >> there seemed nothing about changes to XFS. > > Does it make a difference which of the two kernels creates the file? > > poc > > -- > fedora-list mailing list > fedora-list@redhat.com > To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list > Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines > Well most of the files which exihibit that issue were in /var/run or /var/log. While investigating the latest kernel I had even cleared /var/run still the files created were a few bytes in size but were allocated 512M. I did nothing except rebooting to old kernel and problem went away. My server is running the same services as in the case when the latest kernel was used. So it seems that there is a difference in creation of files by the teo kernels. Vijay -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
On Tue, 2009-02-17 at 23:05 +, Vijay Gill wrote: > Back again with another observation. > > I found out that this issue happens only with > kernel-2.6.27.12-170.2.5.fc10.i686 (the latest one I have). I had a > previous kernel too (kernel-2.6.27.9-159.fc10.i686) and there is no > problem with that. I tried compare the change log of the two rpms and > there seemed nothing about changes to XFS. Does it make a difference which of the two kernels creates the file? poc -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
Back again with another observation. I found out that this issue happens only with kernel-2.6.27.12-170.2.5.fc10.i686 (the latest one I have). I had a previous kernel too (kernel-2.6.27.9-159.fc10.i686) and there is no problem with that. I tried compare the change log of the two rpms and there seemed nothing about changes to XFS. Vijay -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
On Mon, 2009-02-16 at 07:13 +, Vijay Gill wrote: > 2009/2/16 Patrick O'Callaghan : > > On Sun, 2009-02-15 at 23:45 +, Vijay Gill wrote: > >> > That could be because you're using XFS. > >> > > >> > poc > > [...] > >> Thanks but how does that statement explain the behaviour I am seeing? > > > > I'm reaching here, but since XFS uses allocation strategies different > > from the more familiar ext3 system, I wondered if 'du' could in some > > circumstances report larger numbers than "normal". > > > > Apologies if I'm raving. > > > > poc > > > > -- > > fedora-list mailing list > > fedora-list@redhat.com > > To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list > > Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines > > > > You are spot on Pat. XFS allws you to pre-allocate contiguous area of > disk for a file when it is opened for writing data. This is determined > by allocsize parameter during mount and is used foe reducing > fragmentation(fragmentation in Linux? yes siree, it does happen in > Linux too). > > In my case I have provided 1M which is far much less than 512M which > XFS is pre-allocating. This is what caught my eye. It's nice to have one's intuition confirmed once in a while :-) poc -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
2009/2/16 Patrick O'Callaghan : > On Sun, 2009-02-15 at 23:45 +, Vijay Gill wrote: >> > That could be because you're using XFS. >> > >> > poc > [...] >> Thanks but how does that statement explain the behaviour I am seeing? > > I'm reaching here, but since XFS uses allocation strategies different > from the more familiar ext3 system, I wondered if 'du' could in some > circumstances report larger numbers than "normal". > > Apologies if I'm raving. > > poc > > -- > fedora-list mailing list > fedora-list@redhat.com > To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list > Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines > You are spot on Pat. XFS allws you to pre-allocate contiguous area of disk for a file when it is opened for writing data. This is determined by allocsize parameter during mount and is used foe reducing fragmentation(fragmentation in Linux? yes siree, it does happen in Linux too). In my case I have provided 1M which is far much less than 512M which XFS is pre-allocating. This is what caught my eye. Cheers Vijay -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
On Sun, 2009-02-15 at 23:45 +, Vijay Gill wrote: > > That could be because you're using XFS. > > > > poc [...] > Thanks but how does that statement explain the behaviour I am seeing? I'm reaching here, but since XFS uses allocation strategies different from the more familiar ext3 system, I wondered if 'du' could in some circumstances report larger numbers than "normal". Apologies if I'm raving. poc -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
2009/2/15 Patrick O'Callaghan : > On Sun, 2009-02-15 at 23:11 +, Vijay Gill wrote: >> 2009/2/15 Rick : >> > In article <60fdb1ad0902151205v6ef67c07v128f0c88f5895...@mail.gmail.com>, >> > Vijay Gill wrote: >> > >> >>Running du -h tells that the file occupies 512M but ls -l tells that >> >>the file is a lot smaller. >> > >> > Apples and oranges. You get the file *size* with ls and the disk usage >> > with du. They aren't the same thing. >> > >> > -- >> > http://yosemitenews.info/ >> > >> > -- >> > fedora-list mailing list >> > fedora-list@redhat.com >> > To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list >> > Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines >> > >> >> Thanks but I knew that already. I was just curious why 512Mbytes is >> allocated to a file so small? Also I have provided allocsize which is >> definitely not 512M! > > That could be because you're using XFS. > > poc > > -- > fedora-list mailing list > fedora-list@redhat.com > To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list > Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines > Thanks but how does that statement explain the behaviour I am seeing? -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
On Sun, 2009-02-15 at 23:11 +, Vijay Gill wrote: > 2009/2/15 Rick : > > In article <60fdb1ad0902151205v6ef67c07v128f0c88f5895...@mail.gmail.com>, > > Vijay Gill wrote: > > > >>Running du -h tells that the file occupies 512M but ls -l tells that > >>the file is a lot smaller. > > > > Apples and oranges. You get the file *size* with ls and the disk usage > > with du. They aren't the same thing. > > > > -- > > http://yosemitenews.info/ > > > > -- > > fedora-list mailing list > > fedora-list@redhat.com > > To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list > > Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines > > > > Thanks but I knew that already. I was just curious why 512Mbytes is > allocated to a file so small? Also I have provided allocsize which is > definitely not 512M! That could be because you're using XFS. poc -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
2009/2/15 Vijay Gill : > 2009/2/15 Rick : >> In article <60fdb1ad0902151205v6ef67c07v128f0c88f5895...@mail.gmail.com>, >> Vijay Gill wrote: >> >>>Running du -h tells that the file occupies 512M but ls -l tells that >>>the file is a lot smaller. >> >> Apples and oranges. You get the file *size* with ls and the disk usage >> with du. They aren't the same thing. >> >> -- >> http://yosemitenews.info/ >> I found another funny behavior. Notice the sizes of /var/log/samba/log.winbindd-idmap before and after I run xfs_io -c stats on it. Again just looking at stats must not change the space allocated to the file. [r...@aria tmp]# du -sh /var/log/samba/* 0 /var/log/samba/asus.log 0 /var/log/samba/cores 12K /var/log/samba/___10.1.0.151.log 52K /var/log/samba/log.nmbd 4.0K/var/log/samba/log.smbd 0 /var/log/samba/log.wb-ARIA 0 /var/log/samba/log.wb-BUILTIN 4.0K/var/log/samba/log.winbindd 0 /var/log/samba/log.winbindd-dc-connect 512M/var/log/samba/log.winbindd-idmap 56K /var/log/samba/log.winbindd-idmap.old 512M/var/log/samba/nmbd.log 4.0K/var/log/samba/smbd.log 4.0K/var/log/samba/winbindd.log [r...@aria tmp]# xfs_io -c stat /var/log/samba/log.winbindd-idmap fd.path = "/var/log/samba/log.winbindd-idmap" fd.flags = non-sync,non-direct,read-write stat.ino = 3944597 stat.type = regular file stat.size = 403 stat.blocks = 1048576 fsxattr.xflags = 0x0 [--] fsxattr.projid = 0 fsxattr.extsize = 0 fsxattr.nextents = 2 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136 [r...@aria tmp]# du -sh /var/log/samba/* 0 /var/log/samba/asus.log 0 /var/log/samba/cores 12K /var/log/samba/___10.1.0.151.log 52K /var/log/samba/log.nmbd 4.0K/var/log/samba/log.smbd 0 /var/log/samba/log.wb-ARIA 0 /var/log/samba/log.wb-BUILTIN 4.0K/var/log/samba/log.winbindd 0 /var/log/samba/log.winbindd-dc-connect 4.0K/var/log/samba/log.winbindd-idmap 56K /var/log/samba/log.winbindd-idmap.old 512M/var/log/samba/nmbd.log 4.0K/var/log/samba/smbd.log 4.0K/var/log/samba/winbindd.log [r...@aria tmp]# -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: Can somebody explain this? du & ls showing different sizes
2009/2/15 Rick : > In article <60fdb1ad0902151205v6ef67c07v128f0c88f5895...@mail.gmail.com>, > Vijay Gill wrote: > >>Running du -h tells that the file occupies 512M but ls -l tells that >>the file is a lot smaller. > > Apples and oranges. You get the file *size* with ls and the disk usage > with du. They aren't the same thing. > > -- > http://yosemitenews.info/ > > -- > fedora-list mailing list > fedora-list@redhat.com > To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list > Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines > Thanks but I knew that already. I was just curious why 512Mbytes is allocated to a file so small? Also I have provided allocsize which is definitely not 512M! -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
