I've made a few swings at using glusterfs for the php session store for
a heavily-used web app (~6 million pages daily) and I've found time and
again that cpu usage and odd load characteristics cause glusterfs to be
entirely unsuitable for this use case at least given my configuration.
I posted on this earlier, but I'm hoping I can get some input on this as
things are way better than they were but still not good enough. I'm on
v2.0.9 as the 3.0.x series doesn't seem to be fully settled yet, though
feel free to correct me on that.
I have a two-nodes replicate setup and four clients. Configs are below.
What I see is that one brick gets pegged (load avg of 8) and the other
sites much more idle (load avg of 1). The pegged node ends up with high
run queues and i/o blocked processes. CPU usage on the clients for the
glusterfs processes gets pretty high, consuming at least an entire cpu
when not spiking to consume both. I have very high thread counts on the
clients to hopefully avoid thread waits on i/o requests. All six
machines are identical xen instances.
When one of the bricks is down, cpu usage across the board goes way
down, interactivity goes way up, and things seem overall to be a whole
lot better. Why is that? I would think that having two nodes would at
least result in better read rates.
I've gone through various caching schemes and tried readahead,
writebehind, quick-read, and stat-prefetch. I found quick-read caused a
ton of memory consumption and didn't help on performance. I didn't see
much of a change at all with stat-prefetch.
...Any thoughts?
### fsd.vol:
volume sessions
type storage/posix
option directory /var/glusterfs/sessions
option o-direct off
end-volume
volume data
type storage/posix
option directory /var/glusterfs/data
option o-direct off
end-volume
volume locks0
type features/locks
option mandatory-locks on
subvolumes data
end-volume
volume locks1
type features/locks
option mandatory-locks on
subvolumes sessions
end-volume
volume brick0
type performance/io-threads
option thread-count 32 # default is 16
subvolumes locks0
end-volume
volume brick1
type performance/io-threads
option thread-count 32 # default is 16
subvolumes locks1
end-volume
volume server
type protocol/server
option transport-type tcp
option transport.socket.nodelay on
subvolumes brick0 brick1
option auth.addr.brick0.allow ip's...
option auth.addr.brick1.allow ip's...
end-volume
### client.vol (just one connection shown here)
volume glusterfs0-hs
type protocol/client
option transport-type tcp
option remote-host "ip"
option ping-timeout 10
option transport.socket.nodelay on
option remote-subvolume brick1
end-volume
volume glusterfs1-hs
type protocol/client
option transport-type tcp
option remote-host "ip"
option ping-timeout 10server for each request
option transport.socket.nodelay onspeed
option remote-subvolume brick1
end-volume
volume replicated
type cluster/replicate
subvolumes glusterfs0-hs glusterfs1-hs
end-volume
volume iocache
type performance/io-cache
option cache-size 512MB
option cache-timeout 10
subvolumes replicated
end-volume
volume writeback
type performance/write-behind
option cache-size 128MB
option flush-behind off
subvolumes iocache
end-volume
volume iothreads
type performance/io-threads
option thread-count 100
subvolumes writeback
end-volume
--
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
[email protected]
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
