Hello,
Unfortunately, the problem hit me again, just before the holidays…
Just another production system crashed because "no space on the device".
# cat /proc/mounts
rootfs / rootfs rw 0 0
none / tmpfs rw,relatime,size=204800k,mode=755 0 0
/dev/squashfs_p /rootfs_ro squashfs ro,relatime 0 0
none /var aufs rw,relatime,si=20cb810,trunc_xino 0 0
none /usr aufs rw,relatime,si=20cbc10,trunc_xino 0 0
none /srv aufs rw,relatime,si=23c0810,trunc_xino 0 0
none /sbin aufs rw,relatime,si=20b0610,trunc_xino 0 0
none /run aufs rw,relatime,si=23c0a10,trunc_xino 0 0
none /root aufs rw,relatime,si=20b0410,trunc_xino 0 0
none /opt aufs rw,relatime,si=23c0c10,trunc_xino 0 0
none /media aufs rw,relatime,si=20b0210,trunc_xino 0 0
none /lib aufs rw,relatime,si=23c0e10,trunc_xino 0 0
none /home aufs rw,relatime,si=23c0010,trunc_xino 0 0
none /etc aufs rw,relatime,si=20b0e10,trunc_xino 0 0
none /bin aufs rw,relatime,si=20b0a10,trunc_xino 0 0
none /var/run ramfs rw,relatime,size=256k 0 0
none /proc proc rw,nosuid,nodev,noexec,relatime 0 0
none /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
run /run tmpfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=451184k,nr_inodes=112796,mode=755 0 0
none /dev/pts devpts rw,relatime,gid=5,mode=620 0 0
/dev/sda3 /local-config ext4 rw,nosuid,noexec,noatime,commit=30,data=ordered 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
# df /
Filesystem 1K-blocks Used Available Use% Mounted on
none 204800 204800 0 100% /
dmesg contains lots of:
[837130.254973] aufs au_xino_write:416:update_axeos_ce[10384]: I/O Error, write
failed (-5)
[837130.255007] aufs au_xino_do_write:380:update_axeos_ce[10384]: I/O Error,
write failed (4294967268)
[837130.255023] aufs au_xino_do_write:380:update_axeos_ce[10384]: I/O Error,
write failed (4294967268)
[837130.255024] aufs au_xino_write:416:update_axeos_ce[10384]: I/O Error, write
failed (-5)
[837130.259835] aufs au_xino_do_write:380:update_axeos_ce[10384]: I/O Error,
write failed (4294967268)
Trying to manually trunc xino fails:
# mount -o remount,itrunc_xino=0 /etc
mount: No space left on device
But when I have un-mounted /etc and mounted it again (clearing the xino files
for etc)
I was able to truncate the xino files for /var with:
# mount -o remount,itrunc_xino=0 /var
restoring the remaining lost data. So it seems, the automatic xino truncation
failed.
Data from sysfs:
/sys/fs/aufs/si_20b0210/br0: /rw/media=rw
/sys/fs/aufs/si_20b0210/br1: /rootfs_ro/media=rr
/sys/fs/aufs/si_20b0210/xi_path: /rw/media/.aufs.xino
/sys/fs/aufs/si_20b0410/br0: /rw/root=rw
/sys/fs/aufs/si_20b0410/br1: /rootfs_ro/root=rr
/sys/fs/aufs/si_20b0410/xi_path: /rw/root/.aufs.xino
/sys/fs/aufs/si_20b0610/br0: /rw/sbin=rw
/sys/fs/aufs/si_20b0610/br1: /rootfs_ro/sbin=rr
/sys/fs/aufs/si_20b0610/xi_path: /rw/sbin/.aufs.xino
/sys/fs/aufs/si_20b0a10/br0: /rw/bin=rw
/sys/fs/aufs/si_20b0a10/br1: /rootfs_ro/bin=rr
/sys/fs/aufs/si_20b0a10/xi_path: /rw/bin/.aufs.xino
/sys/fs/aufs/si_20b0e10/br0: /rw/etc=rw
/sys/fs/aufs/si_20b0e10/br1: /rootfs_ro/etc=rr
/sys/fs/aufs/si_20b0e10/xi_path: /rw/etc/.aufs.xino
/sys/fs/aufs/si_20cb810/br0: /rw/var=rw
/sys/fs/aufs/si_20cb810/br1: /rootfs_ro/var=rr
/sys/fs/aufs/si_20cb810/xi_path: /rw/var/.aufs.xino
/sys/fs/aufs/si_20cbc10/br0: /rw/usr=rw
/sys/fs/aufs/si_20cbc10/br1: /rootfs_ro/usr=rr
/sys/fs/aufs/si_20cbc10/xi_path: /rw/usr/.aufs.xino
/sys/fs/aufs/si_23c0010/br0: /rw/home=rw
/sys/fs/aufs/si_23c0010/br1: /rootfs_ro/home=rr
/sys/fs/aufs/si_23c0010/xi_path: /rw/home/.aufs.xino
/sys/fs/aufs/si_23c0810/br0: /rw/srv=rw
/sys/fs/aufs/si_23c0810/br1: /rootfs_ro/srv=rr
/sys/fs/aufs/si_23c0810/xi_path: /rw/srv/.aufs.xino
/sys/fs/aufs/si_23c0a10/br0: /rw/run=rw
/sys/fs/aufs/si_23c0a10/br1: /rootfs_ro/run=rr
/sys/fs/aufs/si_23c0a10/xi_path: /rw/run/.aufs.xino
/sys/fs/aufs/si_23c0c10/br0: /rw/opt=rw
/sys/fs/aufs/si_23c0c10/br1: /rootfs_ro/opt=rr
/sys/fs/aufs/si_23c0c10/xi_path: /rw/opt/.aufs.xino
/sys/fs/aufs/si_23c0e10/br0: /rw/lib=rw
/sys/fs/aufs/si_23c0e10/br1: /rootfs_ro/lib=rr
/sys/fs/aufs/si_23c0e10/xi_path: /rw/lib/.aufs.xino
Data from debugfs:
/sys/kernel/debug/aufs/si_20b0a10/xi1: 1, 16x4096 38088
/sys/kernel/debug/aufs/si_20b0a10/xi0: 1, 8x4096 5056
/sys/kernel/debug/aufs/si_20b0a10/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_20b0e10/xi1: 1, 32x4096 37088
/sys/kernel/debug/aufs/si_20b0e10/xi0: 1, 186552x4096 137020716
/sys/kernel/debug/aufs/si_20b0e10/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_23c0010/xi1: 1, 8x4096 4624
/sys/kernel/debug/aufs/si_23c0010/xi0: 1, 16x4096 13748
/sys/kernel/debug/aufs/si_23c0010/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_23c0e10/xi1: 1, 24x4096 41872
/sys/kernel/debug/aufs/si_23c0e10/xi0: 1, 16x4096 4656
/sys/kernel/debug/aufs/si_23c0e10/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_20b0210/xi1: 1, 8x4096 4644
/sys/kernel/debug/aufs/si_20b0210/xi0: 1, 8x4096 4924
/sys/kernel/debug/aufs/si_20b0210/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_23c0c10/xi1: 1, 8x4096 5200
/sys/kernel/debug/aufs/si_23c0c10/xi0: 1, 8x4096 752
/sys/kernel/debug/aufs/si_23c0c10/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_20b0410/xi1: 1, 8x4096 4584
/sys/kernel/debug/aufs/si_20b0410/xi0: 1, 24x4096 61960
/sys/kernel/debug/aufs/si_20b0410/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_23c0a10/xi1: 1, 8x4096 4640
/sys/kernel/debug/aufs/si_23c0a10/xi0: 1, 8x4096 696
/sys/kernel/debug/aufs/si_23c0a10/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_20b0610/xi1: 1, 16x4096 38672
/sys/kernel/debug/aufs/si_20b0610/xi0: 1, 8x4096 4816
/sys/kernel/debug/aufs/si_20b0610/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_23c0810/xi1: 1, 8x4096 4572
/sys/kernel/debug/aufs/si_23c0810/xi0: 1, 16x4096 14796
/sys/kernel/debug/aufs/si_23c0810/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_20cbc10/xi1: 1, 64x4096 33200
/sys/kernel/debug/aufs/si_20cbc10/xi0: 1, 8x4096 4764
/sys/kernel/debug/aufs/si_20cbc10/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_20cb810/xi1: 1, 16x4096 37252
/sys/kernel/debug/aufs/si_20cb810/xi0: 1, 221376x4096 136966220
/sys/kernel/debug/aufs/si_20cb810/xib: 8x4096 4096
From this I can see most of the sapec is used by si_20b0e10 (/etc)
and si_20cb810 (/var) filesystems - these are being written most often.
Any ideas what should I do now to prevent this problem from coming back?
Running 'mount -o remount,itrunc_xino=0' via cron?
Greets,
Jacek
------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
