Hello again!
On Thu, Aug 29, 2013 at 9:35 AM, <[email protected]> wrote:
> Steffen Dettmer:
>> > "mount -o remount /".
>>
>> unfortunately I cannot start mount:
>>
>> root@NOMAD-BLN-test01Xstd:/rw# mount -o remount /
>> mount: error while loading shared libraries: libblkid.so.1: cannot
>
> If these messages are describing about your /rw tmpfs,
In my test case, I create zero files with dd. When the problem
appears, things like "ls" do not work in the directory where the
files were created, but it seems it always works in the parent
directory. I created the files in the root directory. This is no
realistic situation, I never saw many files in /. So I repeated
the test with creating files in /root and this time I was able to
run mount and it solved the problem! Then I repeated the test and
again was unable to invoke mount. I don't know the difference.
> then it means the spaces for the file you removed before
> remounting are not freed yet. Right?
Yes, apparently not.
> One or more processes still are still holding a file descriptor
> corresponding to that file, I am afraid.
I tested with artifical files created by dd, I cannot imagine
someone hold open filedescriptors on them.
> Can you confirm it by "df /rw" between rm and remount?
To avoid I miss telling important information, I include a
detailed copy&paste of my xterm session:
# cd /root/
# n=0; while true; do dd if=/dev/zero of=large.log.$((n++)) bs=1M
2>/dev/null ; done
....wait...wait...wait...
^C
# cd /rw/root
/rw/root# ls -rt|tail
large.log.32758
[...]
large.log.32768
Note: it dd created several 1M files until disk is full, then
0 byte files are created until some limit (number of
free entries in directory?).
At this test, it stopped at 32768 (even when repeating).
root@NOMAD-BLN-test01Xstd:/rw/root# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
rootfs 129273 1369 127904 2% /
udev 128011 276 127735 1% /ro/dev
tmpfs 129273 238 129035 1% /run
/dev/sda1 60800 28394 32406 47% /ro
aufs-tmpfs 129273 1369 127904 2% /rw
aufs 129273 1369 127904 2% /
tmpfs 129273 1 129272 1% /run/lock
tmpfs 129273 327 128946 1% /dev
tmpfs 129273 2 129271 1% /run/shm
/dev/sda3 38160 39 38121 1% /persist
tmpfs 129273 1 129272 1% /tmp
root@NOMAD-BLN-test01Xstd:/rw/root# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 517092 517092 0 100% /
udev 10240 0 10240 0% /ro/dev
tmpfs 103420 176 103244 1% /run
/dev/sda1 956472 547996 359888 61% /ro
aufs-tmpfs 517092 517092 0 100% /rw
aufs 517092 517092 0 100% /
tmpfs 5120 0 5120 0% /run/lock
tmpfs 10240 0 10240 0% /dev
tmpfs 206820 0 206820 0% /run/shm
/dev/sda3 147770 119992 20150 86% /persist
tmpfs 206820 0 206820 0% /tmp
ok, so disk is full.
/rw/root# rm large.log.*
root@NOMAD-BLN-test01Xstd:/rw/root# ls -a
. ..
ok, files deleted.
root@NOMAD-BLN-test01Xstd:/rw/root# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
rootfs 129273 2393 126880 2% /
udev 128011 276 127735 1% /ro/dev
tmpfs 129273 238 129035 1% /run
/dev/sda1 60800 28394 32406 47% /ro
aufs-tmpfs 129273 2393 126880 2% /rw
aufs 129273 2393 126880 2% /
tmpfs 129273 1 129272 1% /run/lock
tmpfs 129273 327 128946 1% /dev
tmpfs 129273 2 129271 1% /run/shm
/dev/sda3 38160 39 38121 1% /persist
tmpfs 129273 1 129272 1% /tmp
root@NOMAD-BLN-test01Xstd:/rw/root# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 505M 505M 0 100% /
udev 10M 0 10M 0% /ro/dev
tmpfs 101M 176K 101M 1% /run
/dev/sda1 935M 536M 352M 61% /ro
aufs-tmpfs 505M 505M 0 100% /rw
aufs 505M 505M 0 100% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 10M 0 10M 0% /dev
tmpfs 202M 0 202M 0% /run/shm
/dev/sda3 145M 120M 17M 88% /persist
tmpfs 202M 0 202M 0% /tmp
disk still full.
root@NOMAD-BLN-test01Xstd:/rw/root# mount -o remount /
root@NOMAD-BLN-test01Xstd:/rw/root# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
rootfs 129273 62 129211 1% /
udev 128011 276 127735 1% /ro/dev
tmpfs 129273 238 129035 1% /run
/dev/disk/b 60800 28394 32406 47% /ro
aufs-tmpfs 129273 62 129211 1% /rw
aufs 129273 62 129211 1% /
tmpfs 129273 1 129272 1% /run/lock
tmpfs 129273 327 128946 1% /dev
tmpfs 129273 2 129271 1% /run/shm
/dev/sda3 38160 39 38121 1% /persist
tmpfs 129273 1 129272 1% /tmp
root@NOMAD-BLN-test01Xstd:/rw/root# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 505M 628K 505M 1% /
udev 10M 0 10M 0% /ro/dev
tmpfs 101M 176K 101M 1% /run
/dev/disk/b 935M 536M 352M 61% /ro
aufs-tmpfs 505M 628K 505M 1% /rw
aufs 505M 628K 505M 1% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 10M 0 10M 0% /dev
tmpfs 202M 0 202M 0% /run/shm
/dev/sda3 145M 121M 17M 88% /persist
tmpfs 202M 0 202M 0% /tmp
Problem fixed! Remount helped, great!
Repeating. Once again the procedure.
# cd /root/
# n=0; while true; do dd if=/dev/zero of=large.log.$((n++)) bs=1M
2>/dev/null ; done
...wait....
^C
# ls
ls: reading directory .: Input/output error
# ls /
bin dev home lib ....
# cd /rw/root
/rw/root# ls -rt|tail
-bash: tail: command not found
root@NOMAD-BLN-test01Xstd:/rw/root# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
rootfs 129273 3922 125351 4% /
udev 128011 276 127735 1% /ro/dev
tmpfs 129273 238 129035 1% /run
/dev/disk/ 60800 28394 32406 47% /ro
aufs-tmpfs 129273 3922 125351 4% /rw
aufs 129273 3922 125351 4% /
tmpfs 129273 1 129272 1% /run/lock
tmpfs 129273 327 128946 1% /dev
tmpfs 129273 2 129271 1% /run/shm
/dev/sda3 38160 39 38121 1% /persist
tmpfs 129273 1 129272 1% /tmp
/rw/root# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 505M 505M 0 100% /
udev 10M 0 10M 0% /ro/dev
tmpfs 101M 176K 101M 1% /run
/dev/disk/ 935M 536M 352M 61% /ro
aufs-tmpfs 505M 505M 0 100% /rw
aufs 505M 505M 0 100% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 10M 0 10M 0% /dev
tmpfs 202M 0 202M 0% /run/shm
/dev/sda3 145M 124M 14M 91% /persist
tmpfs 202M 0 202M 0% /tmp
/rw/root# rm large.log.*
-bash: /bin/rm: No space left on device
/rw/root# rm large.log.0
-bash: /bin/rm: No space left on device
/rw/root# mount -o remount /
mount: error while loading shared libraries: libblkid.so.1: cannot
open shared object file: Error 28
Ohh, you see, this time even unable to delete the files, also not via /rw/:
# rm /rw/large.log.0
-bash: /bin/rm: No space left on device
# rm /root/large.log.0
-bash: /bin/rm: No space left on device
# rm /ro/large.log.0
-bash: /bin/rm: No space left on device
This seems to be different that from I had in yesterdays tests,
there rm was working but did not change free space, this time
even rm was not working.
Because the files are created by dd, I cannot imagine that they
could be open (and rm should work even if they are open of
course).
Note: This time the problem appeared on second "run" of the test case,
but yesterday I had to repeat it many more times (10? 20?).
Normally I would repeat the whole procedure a couple of times,
hoping to reach the situation where rm works but without effect
on dd, but unfortunately I have to work on other tasks with high
priority now, I'm already really late...
> If the cause of those messages is related to aufs XINO files (which are
> not so large), then putting the XINO files in another tmpfs will be
> effective.
> Altenatively, a simple remount before your /rw gets full may be
> effective too. It will prevent aufs to write XINO files at remount-time
> after /rw got full.
Do I understand correctly that calling "mount -o remount /" every
few minutes is recommended?
Of course best is preventing disk from getting full, of course
you are very right - and we have the emergency reboot dealing
with the "must not happen" case of a full file system.
Regards,
Steffen
------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
