Hi Dan - I've seen this problem too. I agree with everything you've
described - seems to happen more quickly on more heavily used volumes,
and a restart fixes it right away. I've also been considering writing a
cronjob to fix this - have you made any progress on this, anything to
report?
I'm running a fairly simple distributed, non-replicated volume across
two servers. What sort of tasks are you using your gluster for? Ours is
for a render farm, so we see a very large number of mounts/unmounts as
render nodes mount various parts of the filesystem. I wonder if this has
anything to do with it; is your use case anything similar?
- brian
On 4/17/12 7:30 PM, Dan Bretherton wrote:
Dear All-
I find that I have to restart glusterd every few days on my servers to
stop NFS performance from becoming unbearably slow. When the problem
occurs, volumes can take several minutes to mount and there are long
delays responding to "ls". Mounting from a different server, i.e.
one not normally used for NFS export, results in normal NFS access
speeds. This doesn't seem to have anything to do with load because it
happens whether or not there is anything running on the compute
servers. Even when the system is mostly idle there are often a lot of
glusterfsd processes running, and on several of the servers I looked
at this evening there is a process called glusterfs using 100% of one
CPU. I can't find anything unusual in nfs.log or
etc-glusterfs-glusterd.vol.log on the servers affected. Restarting
glusterd seems to stop this strange behaviour and make NFS access run
smoothly again, but this usually only lasts for a day or two.
This behaviour is not necessarily related to the length of time since
glusterd was started, but has more to do with the amount of work the
GlusterFS processes on each server have to do. I use a different
server to export each of my 8 different volumes, and the NFS
performance degradation seems to affect the most heavily used volumes
more than the others. I really need to find a solution to this
problem; all I can think of doing is setting up a cron job on each
server to restart glusterd every day, but I am worried about what side
effects that might have. I am using GlusterFS version 3.2.5. All
suggestions would be much appreciated.
Regards,
Dan.
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
