Hello again!

On Thu, Aug 29, 2013 at 9:35 AM,  <sf...@users.sourceforge.net> wrote:
> Steffen Dettmer:
>> >   "mount -o remount /".
>>
>> unfortunately I cannot start mount:
>>
>> root@NOMAD-BLN-test01Xstd:/rw# mount -o remount /
>> mount: error while loading shared libraries: libblkid.so.1: cannot
>
> If these messages are describing about your /rw tmpfs,

In my test case, I create zero files with dd.  When the problem
appears, things like "ls" do not work in the directory where the
files were created, but it seems it always works in the parent
directory. I created the files in the root directory. This is no
realistic situation, I never saw many files in /. So I repeated
the test with creating files in /root and this time I was able to
run mount and it solved the problem! Then I repeated the test and
again was unable to invoke mount. I don't know the difference.

> then it means the spaces for the file you removed before
> remounting are not freed yet. Right?

Yes, apparently not.

> One or more processes still are still holding a file descriptor
> corresponding to that file, I am afraid.

I tested with artifical files created by dd, I cannot imagine
someone hold open filedescriptors on them.

> Can you confirm it by "df /rw" between rm and remount?

To avoid I miss telling important information, I include a
detailed copy&paste of my xterm session:

# cd /root/
# n=0; while true; do dd if=/dev/zero of=large.log.$((n++)) bs=1M
2>/dev/null ; done
....wait...wait...wait...
^C
# cd /rw/root
/rw/root# ls -rt|tail
large.log.32758
 [...]
large.log.32768

   Note: it dd created several 1M files until disk is full, then
         0 byte files are created until some limit (number of
         free entries in directory?).
         At this test, it stopped at 32768 (even when repeating).

root@NOMAD-BLN-test01Xstd:/rw/root# df -i
Filesystem         Inodes IUsed  IFree IUse% Mounted on
rootfs             129273  1369 127904    2% /
udev               128011   276 127735    1% /ro/dev
tmpfs              129273   238 129035    1% /run
/dev/sda1          60800 28394  32406   47% /ro
aufs-tmpfs         129273  1369 127904    2% /rw
aufs               129273  1369 127904    2% /
tmpfs              129273     1 129272    1% /run/lock
tmpfs              129273   327 128946    1% /dev
tmpfs              129273     2 129271    1% /run/shm
/dev/sda3           38160    39  38121    1% /persist
tmpfs              129273     1 129272    1% /tmp

root@NOMAD-BLN-test01Xstd:/rw/root# df
Filesystem      1K-blocks   Used Available Use% Mounted on
rootfs             517092 517092         0 100% /
udev                10240      0     10240   0% /ro/dev
tmpfs              103420    176    103244   1% /run
/dev/sda1          956472 547996    359888  61% /ro
aufs-tmpfs         517092 517092         0 100% /rw
aufs               517092 517092         0 100% /
tmpfs                5120      0      5120   0% /run/lock
tmpfs               10240      0     10240   0% /dev
tmpfs              206820      0    206820   0% /run/shm
/dev/sda3          147770 119992     20150  86% /persist
tmpfs              206820      0    206820   0% /tmp

ok, so disk is full.

/rw/root# rm large.log.*
root@NOMAD-BLN-test01Xstd:/rw/root# ls -a
.  ..

ok, files deleted.

root@NOMAD-BLN-test01Xstd:/rw/root# df -i
Filesystem        Inodes IUsed  IFree IUse% Mounted on
rootfs            129273  2393 126880    2% /
udev              128011   276 127735    1% /ro/dev
tmpfs             129273   238 129035    1% /run
/dev/sda1          60800 28394  32406   47% /ro
aufs-tmpfs        129273  2393 126880    2% /rw
aufs              129273  2393 126880    2% /
tmpfs             129273     1 129272    1% /run/lock
tmpfs             129273   327 128946    1% /dev
tmpfs             129273     2 129271    1% /run/shm
/dev/sda3          38160    39  38121    1% /persist
tmpfs             129273     1 129272    1% /tmp

root@NOMAD-BLN-test01Xstd:/rw/root# df -h
Filesystem        Size  Used Avail Use% Mounted on
rootfs            505M  505M     0 100% /
udev               10M     0   10M   0% /ro/dev
tmpfs             101M  176K  101M   1% /run
/dev/sda1         935M  536M  352M  61% /ro
aufs-tmpfs        505M  505M     0 100% /rw
aufs              505M  505M     0 100% /
tmpfs             5.0M     0  5.0M   0% /run/lock
tmpfs              10M     0   10M   0% /dev
tmpfs             202M     0  202M   0% /run/shm
/dev/sda3         145M  120M   17M  88% /persist
tmpfs             202M     0  202M   0% /tmp

disk still full.

root@NOMAD-BLN-test01Xstd:/rw/root# mount -o remount /

root@NOMAD-BLN-test01Xstd:/rw/root# df -i
Filesystem       Inodes IUsed  IFree IUse% Mounted on
rootfs           129273    62 129211    1% /
udev             128011   276 127735    1% /ro/dev
tmpfs            129273   238 129035    1% /run
/dev/disk/b       60800 28394  32406   47% /ro
aufs-tmpfs       129273    62 129211    1% /rw
aufs             129273    62 129211    1% /
tmpfs            129273     1 129272    1% /run/lock
tmpfs            129273   327 128946    1% /dev
tmpfs            129273     2 129271    1% /run/shm
/dev/sda3         38160    39  38121    1% /persist
tmpfs            129273     1 129272    1% /tmp

root@NOMAD-BLN-test01Xstd:/rw/root# df -h
Filesystem         Size  Used Avail Use% Mounted on
rootfs             505M  628K  505M   1% /
udev                10M     0   10M   0% /ro/dev
tmpfs              101M  176K  101M   1% /run
/dev/disk/b        935M  536M  352M  61% /ro
aufs-tmpfs         505M  628K  505M   1% /rw
aufs               505M  628K  505M   1% /
tmpfs              5.0M     0  5.0M   0% /run/lock
tmpfs               10M     0   10M   0% /dev
tmpfs              202M     0  202M   0% /run/shm
/dev/sda3          145M  121M   17M  88% /persist
tmpfs              202M     0  202M   0% /tmp

Problem fixed! Remount helped, great!

Repeating. Once again the procedure.

# cd /root/
# n=0; while true; do dd if=/dev/zero of=large.log.$((n++)) bs=1M
2>/dev/null ; done
...wait....
^C
# ls
ls: reading directory .: Input/output error
# ls /
bin   dev  home        lib        ....
# cd /rw/root
/rw/root# ls -rt|tail
-bash: tail: command not found

root@NOMAD-BLN-test01Xstd:/rw/root# df -i
Filesystem        Inodes IUsed  IFree IUse% Mounted on
rootfs            129273  3922 125351    4% /
udev              128011   276 127735    1% /ro/dev
tmpfs             129273   238 129035    1% /run
/dev/disk/         60800 28394  32406   47% /ro
aufs-tmpfs        129273  3922 125351    4% /rw
aufs              129273  3922 125351    4% /
tmpfs             129273     1 129272    1% /run/lock
tmpfs             129273   327 128946    1% /dev
tmpfs             129273     2 129271    1% /run/shm
/dev/sda3          38160    39  38121    1% /persist
tmpfs             129273     1 129272    1% /tmp

/rw/root# df -h
Filesystem         Size  Used Avail Use% Mounted on
rootfs             505M  505M     0 100% /
udev                10M     0   10M   0% /ro/dev
tmpfs              101M  176K  101M   1% /run
/dev/disk/         935M  536M  352M  61% /ro
aufs-tmpfs         505M  505M     0 100% /rw
aufs               505M  505M     0 100% /
tmpfs              5.0M     0  5.0M   0% /run/lock
tmpfs               10M     0   10M   0% /dev
tmpfs              202M     0  202M   0% /run/shm
/dev/sda3          145M  124M   14M  91% /persist
tmpfs              202M     0  202M   0% /tmp

/rw/root# rm large.log.*
-bash: /bin/rm: No space left on device
/rw/root# rm large.log.0
-bash: /bin/rm: No space left on device
/rw/root# mount -o remount /
mount: error while loading shared libraries: libblkid.so.1: cannot
open shared object file: Error 28

Ohh, you see, this time even unable to delete the files, also not via /rw/:

# rm /rw/large.log.0
-bash: /bin/rm: No space left on device
# rm /root/large.log.0
-bash: /bin/rm: No space left on device
# rm /ro/large.log.0
-bash: /bin/rm: No space left on device

This seems to be different that from I had in yesterdays tests,
there rm was working but did not change free space, this time
even rm was not working.
Because the files are created by dd, I cannot imagine that they
could be open (and rm should work even if they are open of
course).

Note: This time the problem appeared on second "run" of the test case,
      but yesterday I had to repeat it many more times (10? 20?).

Normally I would repeat the whole procedure a couple of times,
hoping to reach the situation where rm works but without effect
on dd, but unfortunately I have to work on other tasks with high
priority now, I'm already really late...

> If the cause of those messages is related to aufs XINO files (which are
> not so large), then putting the XINO files in another tmpfs will be
> effective.
> Altenatively, a simple remount before your /rw gets full may be
> effective too. It will prevent aufs to write XINO files at remount-time
> after /rw got full.

Do I understand correctly that calling "mount -o remount /" every
few minutes is recommended?

Of course best is preventing disk from getting full, of course
you are very right - and we have the emergency reboot dealing
with the "must not happen" case of a full file system.

Regards,
Steffen

------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk

Reply via email to