Hello Mr Okajima,
> /data/applications/core/image/TESTFILE - On both machines in the ro
branch
> On NodeA I change the file in /data/applications/core/current, so the file
> is copied to /data/share/core/files.
> Via glusterfs the file is also available on NodeB in
/data/share/core/files.
> The problem, glusterfs is mounted via fuse and as I understood fuse isn't
> capable of inotify. So the aufs mount on NodeB don't get the information
of
> the new file on the writable branch and uses the old file from the cache
or
> from the ro branch.
Then how does the replication (NodeA --> NodeB) in glusterfs/fuse work?
Can you always see the latest /data/share/core/files/TESTFILE on NodeB
(bypassing aufs)?
As long as gluseterfs works in userspace and creates the file, the
inotify should work. When and how does the file become visible in NodeB?
Ok, I'm not such a filesystem expert, so I don't know exactly how glusterfs
work. I think it is compareable to NFS, with the difference that it is
distributed and for mounting fuse is used. I don't have one server where the
files are stored, instead I can define different "bricks/volumes" where the
files should be distributed/replicated and glusterfs does the rest.
If you need more detailed information, I can try to figure it out.
What I can tell you is, changing or creating a file on one machine leads to
an instantanious (depending on some factors e.g. network latency) change on
the other machine.
Some tests I did:
One test to compare the performance, was to run a `while(true); do ls -al;
done;` on /data/share/core/files/ and on /data/applications/core/
current on NodeA, while creating files on NodeB.
Without aufs I can't say if the time offset relies on the console output or
on the glusterfs - in other words: The files were directly available.
On the aufs mount it took several seconds.
The other test of inotify:
Setting up an inotifywait on NodeA for /data/share/core/files/ (the
glusterfs mount) and changing files on NodeB. Inotify didn't recognized the
change.
I also set it up on the same node, on the mounted folder, and it recognized
the change.
"It seems that a part of Inotify already works with fuse. You can mount a
filesystem via sshfs and work on it. You can also start some (local) inotify
on the fuse-sshfs mounted filesystem. Then you will get notifications about
all activity there that is done by the local system. But if someone does
changes to this filesystem through another way (directly on the remote host,
or from another host that has remote-mounted the same filesystem, ...), then
you will not notice that via inotify. "
([1]http://sourceforge.net/apps/mediawiki/fuse/index.php?title=Inotify_chang
e_notification)
This describes our situation very well :-) (exchanging sshfs to glusterfs)
---
Again for the understanding:
With glusterfs you have some "servers" where you store the files physically
and some "clients" which can mount the share.
In our case, the server and client is the same machine, so the files are
stored in "/data/store" and this is also mounted to "/data/share".
We are working with the "/data/share" folder, because in the case of larger
files (which are created on another machine), the file is directly
accessable in the mount, and retrieves the content over the network. Because
we are using currently the replication mode, the files are also copied to
the /data/store folder, but this takes more time.
---
So AFAIK inotify don't work for this (right now). How does the udba=reval
recognize changes on the filesystem?
I hope this information helps. If I can do more tests or what ever to figure
out, don't hesitate.
Best regards
Tobias Wilken
References
1.
http://sourceforge.net/apps/mediawiki/fuse/index.php?title=Inotify_change_notification
------------------------------------------------------------------------------
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
