Hello again! On Thu, Aug 29, 2013 at 9:35 AM, <sf...@users.sourceforge.net> wrote: > Steffen Dettmer: >> > "mount -o remount /". >> >> unfortunately I cannot start mount: >> >> root@NOMAD-BLN-test01Xstd:/rw# mount -o remount / >> mount: error while loading shared libraries: libblkid.so.1: cannot > > If these messages are describing about your /rw tmpfs,
In my test case, I create zero files with dd. When the problem appears, things like "ls" do not work in the directory where the files were created, but it seems it always works in the parent directory. I created the files in the root directory. This is no realistic situation, I never saw many files in /. So I repeated the test with creating files in /root and this time I was able to run mount and it solved the problem! Then I repeated the test and again was unable to invoke mount. I don't know the difference. > then it means the spaces for the file you removed before > remounting are not freed yet. Right? Yes, apparently not. > One or more processes still are still holding a file descriptor > corresponding to that file, I am afraid. I tested with artifical files created by dd, I cannot imagine someone hold open filedescriptors on them. > Can you confirm it by "df /rw" between rm and remount? To avoid I miss telling important information, I include a detailed copy&paste of my xterm session: # cd /root/ # n=0; while true; do dd if=/dev/zero of=large.log.$((n++)) bs=1M 2>/dev/null ; done ....wait...wait...wait... ^C # cd /rw/root /rw/root# ls -rt|tail large.log.32758 [...] large.log.32768 Note: it dd created several 1M files until disk is full, then 0 byte files are created until some limit (number of free entries in directory?). At this test, it stopped at 32768 (even when repeating). root@NOMAD-BLN-test01Xstd:/rw/root# df -i Filesystem Inodes IUsed IFree IUse% Mounted on rootfs 129273 1369 127904 2% / udev 128011 276 127735 1% /ro/dev tmpfs 129273 238 129035 1% /run /dev/sda1 60800 28394 32406 47% /ro aufs-tmpfs 129273 1369 127904 2% /rw aufs 129273 1369 127904 2% / tmpfs 129273 1 129272 1% /run/lock tmpfs 129273 327 128946 1% /dev tmpfs 129273 2 129271 1% /run/shm /dev/sda3 38160 39 38121 1% /persist tmpfs 129273 1 129272 1% /tmp root@NOMAD-BLN-test01Xstd:/rw/root# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 517092 517092 0 100% / udev 10240 0 10240 0% /ro/dev tmpfs 103420 176 103244 1% /run /dev/sda1 956472 547996 359888 61% /ro aufs-tmpfs 517092 517092 0 100% /rw aufs 517092 517092 0 100% / tmpfs 5120 0 5120 0% /run/lock tmpfs 10240 0 10240 0% /dev tmpfs 206820 0 206820 0% /run/shm /dev/sda3 147770 119992 20150 86% /persist tmpfs 206820 0 206820 0% /tmp ok, so disk is full. /rw/root# rm large.log.* root@NOMAD-BLN-test01Xstd:/rw/root# ls -a . .. ok, files deleted. root@NOMAD-BLN-test01Xstd:/rw/root# df -i Filesystem Inodes IUsed IFree IUse% Mounted on rootfs 129273 2393 126880 2% / udev 128011 276 127735 1% /ro/dev tmpfs 129273 238 129035 1% /run /dev/sda1 60800 28394 32406 47% /ro aufs-tmpfs 129273 2393 126880 2% /rw aufs 129273 2393 126880 2% / tmpfs 129273 1 129272 1% /run/lock tmpfs 129273 327 128946 1% /dev tmpfs 129273 2 129271 1% /run/shm /dev/sda3 38160 39 38121 1% /persist tmpfs 129273 1 129272 1% /tmp root@NOMAD-BLN-test01Xstd:/rw/root# df -h Filesystem Size Used Avail Use% Mounted on rootfs 505M 505M 0 100% / udev 10M 0 10M 0% /ro/dev tmpfs 101M 176K 101M 1% /run /dev/sda1 935M 536M 352M 61% /ro aufs-tmpfs 505M 505M 0 100% /rw aufs 505M 505M 0 100% / tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 10M 0 10M 0% /dev tmpfs 202M 0 202M 0% /run/shm /dev/sda3 145M 120M 17M 88% /persist tmpfs 202M 0 202M 0% /tmp disk still full. root@NOMAD-BLN-test01Xstd:/rw/root# mount -o remount / root@NOMAD-BLN-test01Xstd:/rw/root# df -i Filesystem Inodes IUsed IFree IUse% Mounted on rootfs 129273 62 129211 1% / udev 128011 276 127735 1% /ro/dev tmpfs 129273 238 129035 1% /run /dev/disk/b 60800 28394 32406 47% /ro aufs-tmpfs 129273 62 129211 1% /rw aufs 129273 62 129211 1% / tmpfs 129273 1 129272 1% /run/lock tmpfs 129273 327 128946 1% /dev tmpfs 129273 2 129271 1% /run/shm /dev/sda3 38160 39 38121 1% /persist tmpfs 129273 1 129272 1% /tmp root@NOMAD-BLN-test01Xstd:/rw/root# df -h Filesystem Size Used Avail Use% Mounted on rootfs 505M 628K 505M 1% / udev 10M 0 10M 0% /ro/dev tmpfs 101M 176K 101M 1% /run /dev/disk/b 935M 536M 352M 61% /ro aufs-tmpfs 505M 628K 505M 1% /rw aufs 505M 628K 505M 1% / tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 10M 0 10M 0% /dev tmpfs 202M 0 202M 0% /run/shm /dev/sda3 145M 121M 17M 88% /persist tmpfs 202M 0 202M 0% /tmp Problem fixed! Remount helped, great! Repeating. Once again the procedure. # cd /root/ # n=0; while true; do dd if=/dev/zero of=large.log.$((n++)) bs=1M 2>/dev/null ; done ...wait.... ^C # ls ls: reading directory .: Input/output error # ls / bin dev home lib .... # cd /rw/root /rw/root# ls -rt|tail -bash: tail: command not found root@NOMAD-BLN-test01Xstd:/rw/root# df -i Filesystem Inodes IUsed IFree IUse% Mounted on rootfs 129273 3922 125351 4% / udev 128011 276 127735 1% /ro/dev tmpfs 129273 238 129035 1% /run /dev/disk/ 60800 28394 32406 47% /ro aufs-tmpfs 129273 3922 125351 4% /rw aufs 129273 3922 125351 4% / tmpfs 129273 1 129272 1% /run/lock tmpfs 129273 327 128946 1% /dev tmpfs 129273 2 129271 1% /run/shm /dev/sda3 38160 39 38121 1% /persist tmpfs 129273 1 129272 1% /tmp /rw/root# df -h Filesystem Size Used Avail Use% Mounted on rootfs 505M 505M 0 100% / udev 10M 0 10M 0% /ro/dev tmpfs 101M 176K 101M 1% /run /dev/disk/ 935M 536M 352M 61% /ro aufs-tmpfs 505M 505M 0 100% /rw aufs 505M 505M 0 100% / tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 10M 0 10M 0% /dev tmpfs 202M 0 202M 0% /run/shm /dev/sda3 145M 124M 14M 91% /persist tmpfs 202M 0 202M 0% /tmp /rw/root# rm large.log.* -bash: /bin/rm: No space left on device /rw/root# rm large.log.0 -bash: /bin/rm: No space left on device /rw/root# mount -o remount / mount: error while loading shared libraries: libblkid.so.1: cannot open shared object file: Error 28 Ohh, you see, this time even unable to delete the files, also not via /rw/: # rm /rw/large.log.0 -bash: /bin/rm: No space left on device # rm /root/large.log.0 -bash: /bin/rm: No space left on device # rm /ro/large.log.0 -bash: /bin/rm: No space left on device This seems to be different that from I had in yesterdays tests, there rm was working but did not change free space, this time even rm was not working. Because the files are created by dd, I cannot imagine that they could be open (and rm should work even if they are open of course). Note: This time the problem appeared on second "run" of the test case, but yesterday I had to repeat it many more times (10? 20?). Normally I would repeat the whole procedure a couple of times, hoping to reach the situation where rm works but without effect on dd, but unfortunately I have to work on other tasks with high priority now, I'm already really late... > If the cause of those messages is related to aufs XINO files (which are > not so large), then putting the XINO files in another tmpfs will be > effective. > Altenatively, a simple remount before your /rw gets full may be > effective too. It will prevent aufs to write XINO files at remount-time > after /rw got full. Do I understand correctly that calling "mount -o remount /" every few minutes is recommended? Of course best is preventing disk from getting full, of course you are very right - and we have the emergency reboot dealing with the "must not happen" case of a full file system. Regards, Steffen ------------------------------------------------------------------------------ Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk