On 21/11/15 14:25, OmegaPhil wrote: > On 20/11/15 23:42, sf...@users.sourceforge.net wrote: >> Hello OmegaPhil, >> >> OmegaPhil: >>> For some time now (across kernel versions, aufs versions, machines, etc) >>> I have noticed kernel memory allocation failures intermittently when >>> something lists a large aufs directory (the most recent example having >>> 20,000 files in it). My aufs usage is to have multiple equally-important >> ::: >> >> For those who has such many files in a single dir, aufs provides >> readdir-in-userspace feature. In other words, the memory allocation is >> done in user-space instead of kernel-space. >> Try enabling CONFIG_AUFS_RDU, rebuild your kernel, and in userspace set >> LD_PRELOAD and LIBAU environment variables. See aufs manual and >> aufs-util manual for details. >> >> >> J. R. Okajima > > > Thanks - have just read this in the manpage - I don't have 'millions of > files' in one directory (although there'll be over a million files in > the whole volume), and the box itself has ~4GB of RAM (2GB used, which > includes the kernel slab) so there shouldn't be any excuse of actually > running out of memory. > > Anyway, I'm happy to look into readdir-in-userspace regardless - > currently I use the stock Debian kernels (CONFIG_AUFS_RDU isn't part of > that source which is already patched to run aufs) so I don't have to > rebuild the kernel, but this should just be recompiling the aufs driver > which is fine. I'll get on with that now, thanks.
libau.so comes with the Debian aufs-tools package which is still in the repos, so I'm using this - currently the rsync init.d script has been edited to export the right LD_PRELOAD and LIBAU values, and I've confirmed the library has been loaded via: ============================ lsof -a -c rsync +D /usr/lib ============================ Looking at kern logs the problem happened twice so far this month and 4 times in October, so a month's test should demonstrate this working.
signature.asc
Description: OpenPGP digital signature
------------------------------------------------------------------------------
