On Nov 14, 2012, at 12:45 PM, Pat Haley wrote:
We have a cluster with 130 compute nodes with an NAS-type
central storage under gluster (3 bricks, ~50TB). When we
run large number of ocean models we can run into bottlenecks
with many jobs trying to write to our central storage.
It was suggested to us that we could also used gluster to
unite the disks on the compute nodes into a single "disk"
in which files would be written locally. Then we could
move the files after the runs were complete in a more
sequential manner (thus avoiding overloading the network).
What was originally suggested (the NUFA policy) has since
been deprecated. What would be the recommended method
of accomplishing our goal in the latest version of Gluster?
And where can we find documentation on it?
We have seen the following links, but would be interested
in any more pointers you may have. Thanks.
http://thr3ads.net/gluster-users/2012/06/1941337-how-to-enable-nufa-in-3.2.6
http://blog.aeste.my/2012/05/15/glusterfs-3-2-updates/
http://www.gluster.org/2012/05/back-door-async-replication/
https://github.com/jdarcy/bypass
s the author of the last two, I guess I should weigh in here. There's a
danger with these kinds of solutions, which is the creation of
conflicts. What
if two clients write to their local copies using bypass? The subsequent
self-heal will fail due to split brain. Sometimes this isn't an issue
because
of the way applications work, or can be avoided by locking the file, but
otherwise it's a real danger.
Moving from replication to distribution, NUFA is deprecated in truth
regardless of the official status. There's no CLI support, and it
lacks many
bug fixes and enhancements - e.g. dynamic reconfiguration, rebalance,
replace-brick - that have been applied to DHT. It builds and appears
to work,
but in my brief tests just now didn't seem to distribute files any
differently
than DHT would.
All is not lost, however. You can get a limited kind of NUFA-like
behavior by
using user-specified layouts. Here's some background:
http://hekafs.org/index.php/2011/04/glusterfs-extended-attributes/
http://hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/
A while back, I added a feature to mark the extended attributes that
we use as
having been set by the user, so that we'll leave them alone when we
rebalance.
What you can do is set these attributes on a directory so that the
local node
"claims" practically all of the consistent-hashing space and new files
will be
placed there but still findable by all of the usual methods and fully
integrated with all the rest of DHT. When your app is done, you can
just set
the layout type back to normal and let DHT recalaculate it (there's a
magic
xattr to do this for a single directory so you don't need to do it
across the
whole volume). If you actually want to migrate the files around
physically,
you could do it the same way we do - check whether the file should be
somewhere else, if so create a copy and rename over the original.
I know that all sounds a bit hackish and cumbersome, but it would work
and not
risk incompatibility with the rest of GlusterFS. I even have some
scripts to
do some similar things, so if you'd like to collaborate with me on
automating
some of this I'd be glad to.
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
