Hey List,
its me again, with a glusterfs/udba performance problem.
The Scenario:
I have two machines, lets call them NodeA and NodeB. Both are are configured
exactly the same.
- /data/applications/core/image is used as a read only aufs branch
- /data/share/core/files is used as a writable aufs branch
and both are mounted to:
- /data/applications/core/current
/data/share is a mounted glusterfs share. So the writeable aufs branch lies
on the shared filesystem.
If I change/override one file on one of the machines, the other machine
doesn't recognize it, or it takes long time.
I think an example explains best:
/data/applications/core/image/TESTFILE - On both machines in the ro branch
On NodeA I change the file in /data/applications/core/current, so the file
is copied to /data/share/core/files.
Via glusterfs the file is also available on NodeB in /data/share/core/files.
The problem, glusterfs is mounted via fuse and as I understood fuse isn't
capable of inotify. So the aufs mount on NodeB don't get the information of
the new file on the writable branch and uses the old file from the cache or
from the ro branch.
I hope this was understandable :-)
Ideas how it could be worked around:
- Reducing the cache (if possible) to nearly zero, but this could be a
performance break in.
- Increasing the lookup time for the writable branch. I don't know how aufs
works in this way
Is this possible in a way or better solutions?
-----
Current work around:
We use glusterfs in a replicated mode, so the machines have a /data/store
folder, too. This is the folder where glusterfs stores the files physically
on each machine. Currently we have an inotify agent watching this folder and
if a file is created/modified we make an `mount -o remount` on the aufs
mount. This works more or less good.
But I only described the simple scenario, there are multiple applications
where each has an aufs mount. So by increasing the applications, I have to
increase the number auf inotify agents and this won't be good. :-)
-----
Best regards
Tobias Wilken
--------------------------
- /proc/mounts (instead of the output of mount(8)):
none /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
none /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev tmpfs rw,relatime,size=10240k,mode=755 0 0
/dev/hda1 / ext2 rw,noatime,nodiratime,errors=remount-ro,acl 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,relatime,mode=755 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
devpts/dev/ptsdevptsrw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
0 0
/dev/hdb1 /data ext3
rw,noatime,nodiratime,errors=remount-ro,acl,data=ordered 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/etc/glusterfs/glusterfs.vol /data/share fuse.glusterfs
rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=13
1072 0 0
none /data/applications/core/current aufs rw,relatime,si=586e167f09164b25 0
0
--------------------------
- /sys/module/aufs/*:
web-3013:~# cat /sys/module/aufs/parameters/
brs nwkq
web-3013:~# cat /sys/module/aufs/parameters/brs
1
web-3013:~# cat /sys/module/aufs/parameters/nwkq
4
--------------------------
- /sys/fs/aufs/* (if you have them)
web-3013:~# cat /sys/fs/aufs/si_586e167f09164b25/
br0 br1 xi_path
web-3013:~# cat /sys/fs/aufs/si_586e167f09164b25/br0
/data/share/core/files=rw
web-3013:~# cat /sys/fs/aufs/si_586e167f09164b25/br1
/data/applications/core/image=ro
web-3013:~# cat /sys/fs/aufs/si_586e167f09164b25/xi_path
/tmp/.aufs.xino
--------------------------
- linux kernel version
web-3013:~# uname -a
Linux web-3013 2.6.30 #8 SMP Mon Oct 26 21:37:24 UTC 2009 x86_64 GNU/Linux
--------------------------
- aufs version which was printed at loading the module or booting the
system, instead of the date you downloaded.
web-3013:~# dmesg |grep aufs
[ 0.273956] aufs 2-standalone.tree-30-20091012
[ 426.806799] aufs test_add:242:mount[2031]: uid/gid/perm
/data/applications/core/image 1001/1001/0777, 1001/1001/0751
--------------------------
- configuration (define/undefine CONFIG_AUFS_xxx)
CONFIG_AUFS_FS=y
CONFIG_AUFS_BRANCH_MAX_127=y
# CONFIG_AUFS_BRANCH_MAX_511 is not set
# CONFIG_AUFS_BRANCH_MAX_1023 is not set
# CONFIG_AUFS_BRANCH_MAX_32767 is not set
CONFIG_AUFS_HINOTIFY=y
# CONFIG_AUFS_SHWH is not set
# CONFIG_AUFS_BR_RAMFS is not set
CONFIG_AUFS_BR_FUSE=y
# CONFIG_AUFS_DEBUG is not set
CONFIG_AUFS_POLL=y
------------------------------------------------------------------------------
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
