Re: AUFS fs unstable once top layer is full
Am 02.11.2015 um 18:14 schrieb sf...@users.sourceforge.net: > There must exist several 'xino[0-9]*' files under > /aufs/si_/, and usually they are larger than xib and xigen. Here are all my files: # tail /sys/kernel/debug/aufs/si_*/* ==> /sys/kernel/debug/aufs/si_e6cc0b3f/plink <== 100 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xi0 <== 1, 1360x4096 725944 ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xi1 <== 1, 8x4096 512 ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xi2 <== 1, 40x4096 17664 ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xi3 <== 1, 32x4096 15180 ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xi4 <== 1, 24x4096 10048 ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xi5 <== 1, 120x4096 61280 ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xi6 <== 1, 64x4096 30544 ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xi7 <== 1, 88x4096 44072 ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xib <== 16x4096 8192 ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xigen <== 288x4096 160100 > And note that the consumed disk space is represented in the form of > 'blocks x block-size'. In your xib case, 64KB is consumed (while the > file size is 8KB). It means 14 blocks are pre-allocated (or had ever > used). > So "xigen" is using 288 * 4096 = about 1,1 Megabytes? Based on all the files listed above, is the total memory consumption in my case (1360+8+40+32+24+12+64+88+16+288)*4096 = nearly 8 MB? Udo --
Re: AUFS fs unstable once top layer is full
Udo Giacomozzi: > ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xi0 <== > 1, 1360x4096 725944 ::: > ==> /sys/kernel/debug/aufs/si_e6cc0b3f/xi5 <== > 1, 120x4096 61280 ::: > So "xigen" is using 288 * 4096 = about 1,1 Megabytes? Exactly. > Based on all the files listed above, is the total memory consumption in > my case (1360+8+40+32+24+12+64+88+16+288)*4096 = nearly 8 MB? Righty right. (But xi5 shows 120 instead of 12). And I called 'xino[0-9]' but it was wrong sorry. You might notice 1360 blocks for xi0 is abnormaly large. If so, I'd suggest you to read my first mail again (and aufs manual) and try truncating XINO. J. R. Okajima --
Re: AUFS fs unstable once top layer is full
Hello Udo,
Udo Giacomozzi:
> Problem: When the tmpfs fills up (eg. `cat /dev/zero > /DUMMY`), then
> the system consistently becomes pratically unusable, meaning that simple
> commands like `find` or `reboot` can't be run anymore ("command not found").
>
> I've also noticed some kernel errors (at least when running for some
> hours in such a state), but they don't come up immediately:
>
> |Oct 31 15:31:01 intermodul kernel: [87752.031289] aufs
> au_xino_do_write:433:ioget[14146]: I/O Error, write failed (-28) Oct 31
> 15:31:01 intermodul kernel: [87752.031301] aufs
> au_xino_write:469:ioget[14146]: I/O Error, write failed (-5) Oct 31
> 15:31:01 intermodul kernel: [87752.050390] aufs
> au_xino_do_write:433:ioget[14148]: I/O Error, write failed (-28) Oct 31
> 15:31:01 intermodul kernel: [87752.050402] aufs
> au_xino_write:469:ioget[14148]: I/O Error, write failed (-5) # df
:::
This situation is something similar to creating a new file where the
filesystem is full. I know you are not creating a file, but for aufs,
accessing a new file means assigning a new inode number, hence expanding
the XINO file. The number '-28' means ENOSPC (No space left on device),
and aufs can do nothing but returning '-5' EIO (Input/output error).
You might not know about XINO files, then I'd suggest you to read aufs
manual especially "External Inode Number Bitmap, Translation Table and
Generation Table (xino)" section.
I guess the most appropriate solution for you will be...
- specify xino=/dev/shm/aufs.xino when mounting aufs
- your df shows /dev/shm has 256MB free, I guess it will be enough
The long story (if you are interested)
- you can confirm the current XINO file path via
/fs/aufs/si_/xi_path.
- also, for the current size of XINO files, see
/aufs/si_/{xib,xino[0-9]*,xigen}
- I guess the path is /root-rw/.aufs.xino, and the consumed blocks is
very large (most of your /root-rw (65MB))
In order to suppres the growing size of XINO files, aufs provides a few
mount options such as
- trunc_xib
- trunc_xino_path=BRANCH | itrunc_xino=INDEX
- trunc_xino
But they are not so effective in some cases. the better solution to
suppres XINO files is to apply tmpfs-idr.patch and vfs-ino.patch against
your kernel. but I am afraid you don't want to do that. So specifying
xino= is a good and effective solution for you I hope.
J. R. Okajima
--
Re: AUFS fs unstable once top layer is full
Hello J. R. Okajima,
thank you ver ymuch for your very helpful information.
I don't have /dev/shm in "init" (when AUFS is set up) so I simply
mounted a tmpfs just for AUFS's "xino" file and it works like a charm!
Great!
After a `find / -mount` my debugfs files show these contents:
xib: 16x4096 8192
xigen: 280x4096 160684
Does this mean that AUFS in my case can consume up to approx. 170kb of
RAM (plus the tmpfs layer) ? I could live very well with that...
Thanks again,
Udo
Am 02.11.2015 um 15:25 schrieb sf...@users.sourceforge.net:
> Hello Udo,
>
> Udo Giacomozzi:
>> Problem: When the tmpfs fills up (eg. `cat /dev/zero > /DUMMY`), then
>> the system consistently becomes pratically unusable, meaning that simple
>> commands like `find` or `reboot` can't be run anymore ("command not found").
>>
>> I've also noticed some kernel errors (at least when running for some
>> hours in such a state), but they don't come up immediately:
>>
>> |Oct 31 15:31:01 intermodul kernel: [87752.031289] aufs
>> au_xino_do_write:433:ioget[14146]: I/O Error, write failed (-28) Oct 31
>> 15:31:01 intermodul kernel: [87752.031301] aufs
>> au_xino_write:469:ioget[14146]: I/O Error, write failed (-5) Oct 31
>> 15:31:01 intermodul kernel: [87752.050390] aufs
>> au_xino_do_write:433:ioget[14148]: I/O Error, write failed (-28) Oct 31
>> 15:31:01 intermodul kernel: [87752.050402] aufs
>> au_xino_write:469:ioget[14148]: I/O Error, write failed (-5) # df
> :::
>
> This situation is something similar to creating a new file where the
> filesystem is full. I know you are not creating a file, but for aufs,
> accessing a new file means assigning a new inode number, hence expanding
> the XINO file. The number '-28' means ENOSPC (No space left on device),
> and aufs can do nothing but returning '-5' EIO (Input/output error).
>
> You might not know about XINO files, then I'd suggest you to read aufs
> manual especially "External Inode Number Bitmap, Translation Table and
> Generation Table (xino)" section.
>
> I guess the most appropriate solution for you will be...
> - specify xino=/dev/shm/aufs.xino when mounting aufs
> - your df shows /dev/shm has 256MB free, I guess it will be enough
>
> The long story (if you are interested)
> - you can confirm the current XINO file path via
>/fs/aufs/si_/xi_path.
> - also, for the current size of XINO files, see
>/aufs/si_/{xib,xino[0-9]*,xigen}
> - I guess the path is /root-rw/.aufs.xino, and the consumed blocks is
>very large (most of your /root-rw (65MB))
>
> In order to suppres the growing size of XINO files, aufs provides a few
> mount options such as
> - trunc_xib
> - trunc_xino_path=BRANCH | itrunc_xino=INDEX
> - trunc_xino
>
> But they are not so effective in some cases. the better solution to
> suppres XINO files is to apply tmpfs-idr.patch and vfs-ino.patch against
> your kernel. but I am afraid you don't want to do that. So specifying
> xino= is a good and effective solution for you I hope.
>
>
> J. R. Okajima
--
Re: AUFS fs unstable once top layer is full
Udo Giacomozzi: > I don't have /dev/shm in "init" (when AUFS is set up) so I simply > mounted a tmpfs just for AUFS's "xino" file and it works like a charm! > Great! Glad to hear that! > After a `find / -mount` my debugfs files show these contents: > > xib: 16x4096 8192 > xigen: 280x4096 160684 > > Does this mean that AUFS in my case can consume up to approx. 170kb of > RAM (plus the tmpfs layer) ? I could live very well with that... There must exist several 'xino[0-9]*' files under /aufs/si_/, and usually they are larger than xib and xigen. And note that the consumed disk space is represented in the form of 'blocks x block-size'. In your xib case, 64KB is consumed (while the file size is 8KB). It means 14 blocks are pre-allocated (or had ever used). J. R. Okajima --
