Re: which files took the space

[Adam Wilson]

On Fri, 04 Mar 2016 08:11:43 +0100 Frédéric Marchal
 wrote:

> On Friday 04 March 2016 07:18:59 Adam Wilson wrote:
> > On Fri, 4 Mar 2016 04:03:01 +1100 Andrew McGlashan
> > 
> >  wrote:
> > > On 4/03/2016 3:07 AM, Adam Wilson wrote:
> > > > On Fri, 4 Mar 2016 03:03:53 +1100 Andrew McGlashan
> > > > 
> > > >  wrote:
> > > >> It also may have been files in the file system, but where
> > > >> another file system mount hides them
> > > > 
> > > > What does this mean? Mounts overlapping and hiding other mounts?
> > > > 
> > > > Explain, please.
> > > 
> > > Yes, this is more likely to happen to the root file system.
> > > 
> > > Say you have a bunch of files in /boot, but for some reason you
> > > have a /boot partition that wasn't mounted when those files were
> > > created  then you mount the /normal/ boot partition over it
> > > and now the other files are now hidden from view, but still
> > > taking up space.
> > 
> > So you're talking about creating files in an unmounted partition,
> > and then mounting it, but since file addition happened when the FS
> > was still in an unmounted state, the new files weren't written to
> > the journal?
> > 
> > Surely in that case the new files would simply not be registered and
> > act as free space (as if they had been deleted)?
> 
> This is not about a caching service. It is about the way the content
> of a mounted disk is made visible to you.
> 
> Have you ever wondered how a path name, on your main hard disk, such
> as /mnt/usb becomes a link to a USB disk known to the system
> as /dev/sdd1?
> 
> When you type
> 
> mount /dev/sdd1 /mnt/usb
> 
> It tells the system that, from now on, the directory named "/mnt/usb"
> becomes the entry point to the file system that sit at "/dev/sdd1".
> 
> Any call such as "ls /mnt/usb" will list the content of the USB disk
> instead of the usual content of /mnt/usb.
> 
> When the device is unmounted, the directory that serves as its mount
> point is restored as a regular directory.
> 
> Now, imagine /mnt/usb is not mounted. It is a directory just like any
> other. If you copy files to /mnt/usb, they will be copied to that
> directory and you have access to them as long as no device is mounted
> under /mnt/usb.

Ah. I see. Thank you for that- I was a little bit unsure as to what was
being referred to, until you came and gave an example.

Re: which files took the space

[Pascal Hambourg]

David Wright a écrit :
> On Sat 05 Mar 2016 at 21:18:55 (+0100), Pascal Hambourg wrote:
>> David Wright a écrit :
>>> You can't create files on an unmounted filesystem.
>> Of course you can, with the adequate tools. For instance mtools for FAT,
>> e2tools for ext2, ntfs-3g (previoulsy ntfsprogs) for NTFS.
> 
> Thank you for taking that sentence completely out of context.

This sentence does not need any context. It is either right or wrong by
itself.

> The tools you mention manipulate devices or images just as any
> program might manipulate a device/file whose structure it is
> familiar with, and not through the operating system.

So what ?
After you mount the filesystem, the file you created with such a tool is
visible. It's just a regular file like any other.

> So those
> "files" are irrelevant to a discussion of mount points and
> visibility

Your sentence was as irrelevant.

> (I wasn't aware that ntfs-3g worked that way: I thought it just
> mounted NTFS filesystems like any other.)

Actually the tools which were previously packaged in ntfsprogs have been
included into ntfs-3g, but I don't think they are used by the ntfs-3g
filesystem.

Re: which files took the space

[David Wright]

On Sat 05 Mar 2016 at 21:18:55 (+0100), Pascal Hambourg wrote:
> David Wright a écrit :
> > 
> > You can't create files on an unmounted filesystem.
> 
> Of course you can, with the adequate tools. For instance mtools for FAT,
> e2tools for ext2, ntfs-3g (previoulsy ntfsprogs) for NTFS.

Thank you for taking that sentence completely out of context.
Why, you even managed to remove the parentheses!

The tools you mention manipulate devices or images just as any
program might manipulate a device/file whose structure it is
familiar with, and not through the operating system. So those
"files" are irrelevant to a discussion of mount points and
visibility because they're not accessed through the OS's file
access methods. To the OS the tools are just programs manipulating
their data, and to the OS those data are not files.

(I wasn't aware that ntfs-3g worked that way: I thought it just
mounted NTFS filesystems like any other.)

Cheers,
David.

Re: which files took the space

[Pascal Hambourg]

David Wright a écrit :
> 
> You can't create files on an unmounted filesystem.

Of course you can, with the adequate tools. For instance mtools for FAT,
e2tools for ext2, ntfs-3g (previoulsy ntfsprogs) for NTFS.

Re: which files took the space

[Pascal Hambourg]

to...@tuxteam.de a écrit :
> 
> It was perceived as an artificial limitation which has no place in
> the kernel. Actually, I can conceive use cases for mount shadowing
> the content of a directory.

Typically : /dev

Re: which files took the space

[Rick Thomas]

On Mar 4, 2016, at 5:39 AM, Gene Heskett  wrote:

> On Friday 04 March 2016 03:39:03 jdd wrote:
> 
>> Le 04/03/2016 08:38, to...@tuxteam.de a écrit :
>>> If memory serves, long, long time ago, "mount" (the system call)
>>> refused to comply unless the directory was empty (kernel 2.6.mumble;
>>> so long ago. Expect bit flips and glitches in my wetware, yadda).
>> 
>> if it existed, it's much older than that, never seen it since 1997...
> 
> Me either, and I go back to '97 myself in linux usage.
> 
>> man mount:
>> 
>> This  tells  the  kernel  to attach the filesystem found on device
>> (which is of type type) at the directory dir.  The previous contents
>> (if any) and owner and mode of dir become  invisible,  and  as  long 
>> as this  filesystem  remains mounted, the pathname dir refers to the
>> root of the filesystem on device."

This behavior (mounted filesystems hiding the files in the underlying 
directory) goes all the way back to Bell Labs UNIX(tm) v7 (at least -- I never 
had a chance to use v6 or earlier) in the 1970's.  It's in the POSIX standard.

Rick

Re: which files took the space

[Jörg-Volker Peetz]

The list seems nowadays to have a very short memory ;-)
A similar problem was discussed last time hardly a month ago:
https://lists.debian.org/debian-user/2016/02/threads.html#00267

So in order to avoid the pit you fall into you could've used
a command like

  du /home -hx --max-depth=1

Regards,
jvp.

Re: which files took the space

[Gene Heskett]

On Friday 04 March 2016 03:39:03 jdd wrote:

> Le 04/03/2016 08:38, to...@tuxteam.de a écrit :
> > If memory serves, long, long time ago, "mount" (the system call)
> > refused to comply unless the directory was empty (kernel 2.6.mumble;
> > so long ago. Expect bit flips and glitches in my wetware, yadda).
>
> if it existed, it's much older than that, never seen it since 1997...

Me either, and I go back to '97 myself in linux usage.

> man mount:
>
> This  tells  the  kernel  to attach the filesystem found on device
> (which is of type type) at the directory dir.  The previous contents
> (if any) and owner and mode of dir become  invisible,  and  as  long 
> as this  filesystem  remains mounted, the pathname dir refers to the
> root of the filesystem on device."
>
>
> jdd


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: which files took the space

[jdd]

Le 04/03/2016 08:38, to...@tuxteam.de a écrit :


If memory serves, long, long time ago, "mount" (the system call)
refused to comply unless the directory was empty (kernel 2.6.mumble;
so long ago. Expect bit flips and glitches in my wetware, yadda).


if it existed, it's much older than that, never seen it since 1997...

man mount:

This  tells  the  kernel  to attach the filesystem found on device 
(which is of type type) at the directory dir.  The previous contents (if 
any) and owner and mode of dir become  invisible,  and  as  long  as 
this  filesystem  remains mounted, the pathname dir refers to the root 
of the filesystem on device."



jdd

Re: which files took the space

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Thu, Mar 03, 2016 at 11:17:19PM -0500, Cindy-Sue Causey wrote:
> On 3/3/16, Andrew McGlashan  wrote:
> >
> > On 4/03/2016 3:07 AM, Adam Wilson wrote:
> >> On Fri, 4 Mar 2016 03:03:53 +1100 Andrew McGlashan
> >>  wrote:
> >>> It also may have been files in the file system, but where another file
> >>> system mount hides them
> >>
> >> What does this mean? Mounts overlapping and hiding other mounts?
> >>
> >> Explain, please.
> >
> > Yes, this is more likely to happen to the root file system.
> >
> > Say you have a bunch of files in /boot, but for some reason you have a
> > /boot partition that wasn't mounted when those files were created 
> > then you mount the /normal/ boot partition over it and now the other
> > files are now hidden from view, but still taking up space.
> 
> 
> Is that behavior as designed and thus expected, or is it a glitch?

If memory serves, long, long time ago, "mount" (the system call)
refused to comply unless the directory was empty (kernel 2.6.mumble;
so long ago. Expect bit flips and glitches in my wetware, yadda).

It was perceived as an artificial limitation which has no place in
the kernel. Actually, I can conceive use cases for mount shadowing
the content of a directory.

> My brain's thinking it should either complain and refuse to continue
> else obliterate and replace.

Nothing gets obliterated. Only shadowed...

> To me it would be... safer that it halt and complain rather than
> destroy, but all that shows is that I most likely just don't
> understand the function.
> 
> Do (and/or should) the original files "reappear" later?

They do -- at unmount. That's where the "missing" space is used,
after all :-)

> Guess I'm just thinking out loud again mostly because I actually
> understand the circumstance as presented. :)

Maling lists are just the hum of many people thinking loud. Keep doing
it :-)

regards
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlbZO3sACgkQBcgs9XrR2kYwNgCfdoVhuenZV6/wf+hGj2KkOIVd
KYsAniD1kTT85SjV1BTpGhR4cghb9yr8
=RjOC
-END PGP SIGNATURE-

Re: which files took the space

[jdd]

Le 04/03/2016 08:11, Frédéric Marchal a écrit :


It can happen with /boot or /home. Mistakenly copying files to an unmounted
drive copies the files to the underlying file system, in the directory that
usually serves as the mount point for the external file system. Once /boot or
/home is mounted, you don't see the files but they are still taking space on
the disk.



I have recently seen such thing happening, due to faulty script 
expecting a usb mounted disk that was not always there. A rsync in such 
situation can fill a disk


jdd

Re: which files took the space

[Frédéric Marchal]

On Friday 04 March 2016 07:18:59 Adam Wilson wrote:
> On Fri, 4 Mar 2016 04:03:01 +1100 Andrew McGlashan
> 
>  wrote:
> > On 4/03/2016 3:07 AM, Adam Wilson wrote:
> > > On Fri, 4 Mar 2016 03:03:53 +1100 Andrew McGlashan
> > > 
> > >  wrote:
> > >> It also may have been files in the file system, but where another
> > >> file system mount hides them
> > > 
> > > What does this mean? Mounts overlapping and hiding other mounts?
> > > 
> > > Explain, please.
> > 
> > Yes, this is more likely to happen to the root file system.
> > 
> > Say you have a bunch of files in /boot, but for some reason you have a
> > /boot partition that wasn't mounted when those files were created 
> > then you mount the /normal/ boot partition over it and now the other
> > files are now hidden from view, but still taking up space.
> 
> So you're talking about creating files in an unmounted partition, and
> then mounting it, but since file addition happened when the FS was
> still in an unmounted state, the new files weren't written to the
> journal?
> 
> Surely in that case the new files would simply not be registered and
> act as free space (as if they had been deleted)?

This is not about a caching service. It is about the way the content of a 
mounted disk is made visible to you.

Have you ever wondered how a path name, on your main hard disk, such as 
/mnt/usb becomes a link to a USB disk known to the system as /dev/sdd1?

When you type

mount /dev/sdd1 /mnt/usb

It tells the system that, from now on, the directory named "/mnt/usb" becomes 
the entry point to the file system that sit at "/dev/sdd1".

Any call such as "ls /mnt/usb" will list the content of the USB disk instead 
of the usual content of /mnt/usb.

When the device is unmounted, the directory that serves as its mount point is 
restored as a regular directory.

Now, imagine /mnt/usb is not mounted. It is a directory just like any other. 
If you copy files to /mnt/usb, they will be copied to that directory and you 
have access to them as long as no device is mounted under /mnt/usb.

Then, you mount the USB disk to /mnt/usb. Now, the files are still on the hard 
disk (let's say /dev/sda1) but the path /mnt/usb now lists the content of the 
USB disk on /dev/sdd1.

It can happen with /boot or /home. Mistakenly copying files to an unmounted 
drive copies the files to the underlying file system, in the directory that 
usually serves as the mount point for the external file system. Once /boot or 
/home is mounted, you don't see the files but they are still taking space on 
the disk.

Hope it helps,

Frederic

Re: which files took the space

[Gene Heskett]

On Thursday 03 March 2016 23:18:59 Adam Wilson wrote:

> On Fri, 4 Mar 2016 04:03:01 +1100 Andrew McGlashan
>
>  wrote:
> > On 4/03/2016 3:07 AM, Adam Wilson wrote:
> > > On Fri, 4 Mar 2016 03:03:53 +1100 Andrew McGlashan
> > >
> > >  wrote:
> > >> It also may have been files in the file system, but where another
> > >> file system mount hides them
> > >
> > > What does this mean? Mounts overlapping and hiding other mounts?
> > >
> > > Explain, please.
> >
> > Yes, this is more likely to happen to the root file system.
> >
> > Say you have a bunch of files in /boot, but for some reason you have
> > a /boot partition that wasn't mounted when those files were created
> >  then you mount the /normal/ boot partition over it and now the
> > other files are now hidden from view, but still taking up space.
>
> So you're talking about creating files in an unmounted partition, and
> then mounting it, but since file addition happened when the FS was
> still in an unmounted state, the new files weren't written to the
> journal?
>
> Surely in that case the new files would simply not be registered and
> act as free space (as if they had been deleted)?

No, the mount point, by whatever name exists as a directory on the hard 
drive, and creating files in that mount point actually creates the files 
and they take up normal space on the drive in that directory.  You can 
then mount another drive or partition on top of that mount point, which 
hides the files, but they are still there, taking up space.

If you need those files to be accessable at that mount point in normal 
operation, you must create a different mount point, umount that storage 
from that mount point and mount it to the new different point.  You will 
now see that the files are there, and you can use some file utility (I'm 
old old school, so mc is the obvious choice) to move them to the storage 
mounted elsewhere temporarily. When that directory is empty, you umount 
that storage from its temporary location and remount it back where it 
will normally live. Now those files really are there.  Problem solved 
and you have your drive space back too.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: which files took the space

[David Wright]

On Fri 04 Mar 2016 at 07:18:59 (+0300), Adam Wilson wrote:
> On Fri, 4 Mar 2016 04:03:01 +1100 Andrew McGlashan
>  wrote:
> 
> > On 4/03/2016 3:07 AM, Adam Wilson wrote:
> > > On Fri, 4 Mar 2016 03:03:53 +1100 Andrew McGlashan
> > >  wrote:
> > >> It also may have been files in the file system, but where another
> > >> file system mount hides them
> > > 
> > > What does this mean? Mounts overlapping and hiding other mounts?
> > > 
> > > Explain, please.
> > 
> > Yes, this is more likely to happen to the root file system.
> > 
> > Say you have a bunch of files in /boot, but for some reason you have a
> > /boot partition that wasn't mounted when those files were created 
> > then you mount the /normal/ boot partition over it and now the other
> > files are now hidden from view, but still taking up space.
> 
> So you're talking about creating files in an unmounted partition, and
> then mounting it, but since file addition happened when the FS was
> still in an unmounted state, the new files weren't written to the
> journal?
> 
> Surely in that case the new files would simply not be registered and
> act as free space (as if they had been deleted)?

No, the other way around. (You can't create files on an unmounted filesystem.)

1. A directory contains some files.
2. You mount a filesystem onto that directory. (You obviously don't
   realise that this, while unusual, is a perfectly well-defined
   operation. †)
3. The directory now contains the top-level files in the filesystem.
   The files that were in the directory before are inaccessible,
   hidden under the mounted filesystem.
4. You unmount the filesystem. Now you can see the original files again.

While the files are hidden, they still occupy the space.

† Take a look at the first three paragraphs of   man mount.

Cheers,
David.

Re: which files took the space

[David Wright]

On Thu 03 Mar 2016 at 23:17:19 (-0500), Cindy-Sue Causey wrote:
> On 3/3/16, Andrew McGlashan  wrote:
> >
> > On 4/03/2016 3:07 AM, Adam Wilson wrote:
> >> On Fri, 4 Mar 2016 03:03:53 +1100 Andrew McGlashan
> >>  wrote:
> >>> It also may have been files in the file system, but where another file
> >>> system mount hides them
> >>
> >> What does this mean? Mounts overlapping and hiding other mounts?
> >>
> >> Explain, please.
> >
> > Yes, this is more likely to happen to the root file system.
> >
> > Say you have a bunch of files in /boot, but for some reason you have a
> > /boot partition that wasn't mounted when those files were created 
> > then you mount the /normal/ boot partition over it and now the other
> > files are now hidden from view, but still taking up space.
> 
> 
> Is that behavior as designed and thus expected, or is it a glitch?

It is as defined by mount's man page.

> My brain's thinking it should either complain and refuse to continue
> else obliterate and replace.

Typically a mount point is an empty directory. However, this does not
_have to be_ the case.

> To me it would be... safer that it halt and complain rather than
> destroy, but all that shows is that I most likely just don't
> understand the function.

Nothing is destroyed, but just inaccessible or "hidden".

> Do (and/or should) the original files "reappear" later?

As soon as you unmount the device that you mounted on that directory.

Cheers,
David.

Re: which files took the space

[Adam Wilson]

On Fri, 4 Mar 2016 04:03:01 +1100 Andrew McGlashan
 wrote:

> 
> 
> On 4/03/2016 3:07 AM, Adam Wilson wrote:
> > On Fri, 4 Mar 2016 03:03:53 +1100 Andrew McGlashan
> >  wrote:
> >> It also may have been files in the file system, but where another
> >> file system mount hides them
> > 
> > What does this mean? Mounts overlapping and hiding other mounts?
> > 
> > Explain, please.
> 
> Yes, this is more likely to happen to the root file system.
> 
> Say you have a bunch of files in /boot, but for some reason you have a
> /boot partition that wasn't mounted when those files were created 
> then you mount the /normal/ boot partition over it and now the other
> files are now hidden from view, but still taking up space.

So you're talking about creating files in an unmounted partition, and
then mounting it, but since file addition happened when the FS was
still in an unmounted state, the new files weren't written to the
journal?

Surely in that case the new files would simply not be registered and
act as free space (as if they had been deleted)?

Re: which files took the space

[Cindy-Sue Causey]

On 3/3/16, Andrew McGlashan  wrote:
>
> On 4/03/2016 3:07 AM, Adam Wilson wrote:
>> On Fri, 4 Mar 2016 03:03:53 +1100 Andrew McGlashan
>>  wrote:
>>> It also may have been files in the file system, but where another file
>>> system mount hides them
>>
>> What does this mean? Mounts overlapping and hiding other mounts?
>>
>> Explain, please.
>
> Yes, this is more likely to happen to the root file system.
>
> Say you have a bunch of files in /boot, but for some reason you have a
> /boot partition that wasn't mounted when those files were created 
> then you mount the /normal/ boot partition over it and now the other
> files are now hidden from view, but still taking up space.


Is that behavior as designed and thus expected, or is it a glitch?

My brain's thinking it should either complain and refuse to continue
else obliterate and replace.

To me it would be... safer that it halt and complain rather than
destroy, but all that shows is that I most likely just don't
understand the function.

Do (and/or should) the original files "reappear" later?

Guess I'm just thinking out loud again mostly because I actually
understand the circumstance as presented. :)

Cindy :)

-- 
Cindy-Sue Causey
Talking Rock, Pickens County, Georgia, USA

* runs with duct tape *

Re: which files took the space

[Andrew McGlashan]

On 4/03/2016 3:07 AM, Adam Wilson wrote:
> On Fri, 4 Mar 2016 03:03:53 +1100 Andrew McGlashan
>  wrote:
>> It also may have been files in the file system, but where another file
>> system mount hides them
> 
> What does this mean? Mounts overlapping and hiding other mounts?
> 
> Explain, please.

Yes, this is more likely to happen to the root file system.

Say you have a bunch of files in /boot, but for some reason you have a
/boot partition that wasn't mounted when those files were created 
then you mount the /normal/ boot partition over it and now the other
files are now hidden from view, but still taking up space.

Cheers
A.

Re: which files took the space

[jdd]

Le 03/03/2016 17:03, Andrew McGlashan a écrit :



On 3/03/2016 1:51 PM, lina wrote:

I figured out, there are so many hidden files.


It also may have been files in the file system, but where another file
system mount hides them

A.

or file in use deleted but not released... like virtual machines or 
mounted isos...


jdd

Re: which files took the space

[Adam Wilson]

On Fri, 4 Mar 2016 03:03:53 +1100 Andrew McGlashan
 wrote:

> 
> 
> On 3/03/2016 1:51 PM, lina wrote:
> > I figured out, there are so many hidden files.
> 
> It also may have been files in the file system, but where another file
> system mount hides them

What does this mean? Mounts overlapping and hiding other mounts?

Explain, please.

Re: which files took the space

[Andrew McGlashan]

On 3/03/2016 1:51 PM, lina wrote:
> I figured out, there are so many hidden files.

It also may have been files in the file system, but where another file
system mount hides them

A.

Re: which files took the space

[Jochen Spieker]

lina:
> 
> how do I know which files mounted on the /home.

In the future you can install and use 'ncdu'. It is a nice terminal
program that calculates directory sizes and allows you to easily see
what occupies the most disk space.

J.
-- 
I like my Toyota RAV4 because of the commanding view of the traffic
jams.
[Agree]   [Disagree]
 


signature.asc
Description: Digital signature

Re: which files took the space

[lina]

I figured out, there are so many hidden files.

Thanks,

On Thu, Mar 3, 2016 at 10:38 AM, lina  wrote:
> Hi,
>
> I did a simple check:
>
> $ df -h
> Filesystem  Size  Used Avail Use% Mounted on
> /dev/sda5   658M  408M  217M  66% /
> /dev/sda4   213M   64M  139M  32% /boot
> /dev/sda714G   13G  593M  96% /home
>
>
> /home
>
> 40Mbin
> 502MCytoscapeConfiguration
> 381MDesktop
> 896MDocuments
> 42Mdwhelper
> 12Ketc
> 2.4Minclude
> 149Mlib
> 1.8MMaildir
> 32KNotebooks
> 3.0Mperl5
> 12Mshare
> 1.3Gsrc
> 13Mtexmf
>
>
> Once added it up, it won't be 13G,
>
> how do I know which files mounted on the /home.
>
> I googled a while, and also reboot the system. still the same.
>
> Thanks for your suggestions,

which files took the space

[lina]

Hi,

I did a simple check:

$ df -h
Filesystem  Size  Used Avail Use% Mounted on
/dev/sda5   658M  408M  217M  66% /
/dev/sda4   213M   64M  139M  32% /boot
/dev/sda714G   13G  593M  96% /home


/home

40Mbin
502MCytoscapeConfiguration
381MDesktop
896MDocuments
42Mdwhelper
12Ketc
2.4Minclude
149Mlib
1.8MMaildir
32KNotebooks
3.0Mperl5
12Mshare
1.3Gsrc
13Mtexmf


Once added it up, it won't be 13G,

how do I know which files mounted on the /home.

I googled a while, and also reboot the system. still the same.

Thanks for your suggestions,