Hi, I recently switched from unionfs-fuse to aufs for a diskless installation using Fedora and noticed some strange behavior while experimenting with it. Let me describe our setup first: We have one server which holds the root filesystem of a Fedora 17 installation and exports it via NFS (3). It also exports a folder containing different configuration groups where each folder contains that parts of a filesystem which need to be changed from the root installation (mostly files in /etc). Then we have about 100 clients which all mount the root file system via NFS from this server. We now use aufs (or unionfs-fuse before that) to a) make the root file system writeable (using tmpfs as upper branch) and b) stack the specific configuration group for each computer (which is selected using dhcp options) on top of the root file system. In order to make that work, we added a script to initrd which does all the magic. Now the strange behavior is as follows: If I add a break just before the initrd executes switch_root (the one from util-linux, not from busybox) every thing looks normal, which means that the important parts of /proc/mounts look like this (/sysroot is the new root file system) proc /proc proc rw,nosuid,nodev,noexec,realtime 0 0 1.2.3.4:/diskless/F17/root /sysroot nfs ro,realtime,vers=3 [..] none /rw tmpfs rw,realtime,mode=755 [..] 1.2.3.4:/diskless/F17/groups /groups nfs ro,realtime,vers=3 [..] none /sysroot aufs rw,realtime [..] The mount command looks like this: mount -t aufs -o br=/rw=rw:/groups/conf_fglrx-tft=ro:/groups/conf_i09=ro:/sysroot=ro -o xino=/rw/.aufs.xino -o udba=notify none /sysroot So the problem is this: After switch_root is executed and the system init process starts, /proc is gone. Normally switch_root moves /sys, /dev and /proc to the new root file system and deletes the contents of the initrd but when using aufs, /dev and /sys appear like expected but /proc is missing (which means, that the folder is just empty). The really strange thing about it is that is does not happen every time but only in about 8/10 times. The times it didn't happen, I always added a break point to a recovery shell just before switch_root, looked around in the file system and just resumed the booting process. I wasn't able to figure out what exactly caused it to work, but it seems to be some kind of race condition, because some waiting was needed every time. I also tried different combinations of moving /proc to /sysroot/proc before switch_root and I also tried using chroot instead of switch_root but it always resulted in the same behavior. But what I did figure out is, how to avoid this problem completely: After issuing the aufs mount command, do the following: rmdir /sysroot/proc mkdir /sysroot/proc Then everything works without problems. Any idea what could be the problem here? Anyway, after I figured out how to work around this, everything booted up fine, but I noticed another strange behavior: after some time, aufs starts to copy-up a large amount of different binaries and libraries even though none of them were modified. At first, in a freshly booted system, everything looks normal (I did mount --move /rw /sysroot/rw, to see which files are copied up): none 3,9G 28M 3,9G 1% /rw Then, after about 30-60 Minutes (it varies each time), aufs starts copying up a large portion of /usr/lib, /usr/lib64 and /usr/bin, until a total of about 1.8 GB of files are copied up to /rw none 3,9G 2,1G 1,9G 54% /rw The output of du -shc /rw/* 0 /rw/bin 1,2M /rw/etc 0 /rw/lib 0 /rw/lib64 0 /rw/proc 8,0K /rw/root 0 /rw/rw 0 /rw/sbin 24K /rw/tmp 1,7G /rw/usr 408M /rw/var 2,1G total Output of du -shc /rw/usr/* 619M /rw/usr/bin 29M /rw/usr/lib 892M /rw/usr/lib64 117M /rw/usr/libexec 31M /rw/usr/sbin 6,0M /rw/usr/share 1,7G total So, is there any way to figure out why aufs is copying up all these libraries? As mentioned, we are using Fedora 17 with kernel version 3.4 and the most recent release of aufs3 (3.4-20120611) and aufs-utils. grep AUFS* config-3.4.3-1.aufs.fc17.x86_64 CONFIG_AUFS_FS=y CONFIG_AUFS_BRANCH_MAX_127=y # CONFIG_AUFS_BRANCH_MAX_511 is not set # CONFIG_AUFS_BRANCH_MAX_1023 is not set # CONFIG_AUFS_BRANCH_MAX_32767 is not set CONFIG_AUFS_SBILIST=y CONFIG_AUFS_HNOTIFY=y CONFIG_AUFS_HFSNOTIFY=y # CONFIG_AUFS_EXPORT is not set CONFIG_AUFS_RDU=y CONFIG_AUFS_PROC_MAP=y CONFIG_AUFS_SP_IATTR=y CONFIG_AUFS_SHWH=y # CONFIG_AUFS_BR_RAMFS is not set # CONFIG_AUFS_BR_FUSE is not set CONFIG_AUFS_BR_HFSPLUS=y CONFIG_AUFS_BDEV_LOOP=y CONFIG_AUFS_DEBUG=y CONFIG_AUFS_MAGIC_SYSRQ=y cat /sys/fs/aufs/si_a8089d1e593ff505/* /rw=rw /groups/conf_fglrx-tft=ro /groups/conf_i09=ro /sysroot=ro /rw/.aufs.xino Thank you for your time, Jochen Ril
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
