On 04/18/2012 01:48 PM, [email protected] wrote:
Date: Tue, 17 Apr 2012 19:06:31 -0500 (CDT)
From: Gerald Brandt<[email protected]>
Subject: Re: [Gluster-users] Frequent glusterd restarts needed to
avoid NFS performance degradation
To: Dan Bretherton<[email protected]>
Cc: gluster-users<[email protected]>
Message-ID:<22749685.104.1334707572319.JavaMail.gbr@thinkpad>
Content-Type: text/plain; charset=utf-8
Hi,
----- Original Message -----
> Dear All-
> I find that I have to restart glusterd every few days on my servers
> to
> stop NFS performance from becoming unbearably slow. When the problem
> occurs, volumes can take several minutes to mount and there are long
> delays responding to "ls". Mounting from a different server, i.e.
> one
> not normally used for NFS export, results in normal NFS access
> speeds.
> This doesn't seem to have anything to do with load because it happens
> whether or not there is anything running on the compute servers.
> Even
> when the system is mostly idle there are often a lot of glusterfsd
> processes running, and on several of the servers I looked at this
> evening there is a process called glusterfs using 100% of one CPU. I
> can't find anything unusual in nfs.log or
> etc-glusterfs-glusterd.vol.log
> on the servers affected. Restarting glusterd seems to stop this
> strange
> behaviour and make NFS access run smoothly again, but this usually
> only
> lasts for a day or two.
>
> This behaviour is not necessarily related to the length of time since
> glusterd was started, but has more to do with the amount of work the
> GlusterFS processes on each server have to do. I use a different
> server
> to export each of my 8 different volumes, and the NFS performance
> degradation seems to affect the most heavily used volumes more than
> the
> others. I really need to find a solution to this problem; all I can
> think of doing is setting up a cron job on each server to restart
> glusterd every day, but I am worried about what side effects that
> might
> have. I am using GlusterFS version 3.2.5. All suggestions would be
> much appreciated.
>
> Regards,
> Dan.
I run GlusterFS 3.2.5 and only access is via NFS. I'm running Citrix XenServer
with about 23 VM's off of it. I haven't seen any degradation at all.
One thing I don't have is replication or anything else set up. The server is
ready to replicate, but I'm waiting for 3.3
Gerald
Hello Gerald,
Thanks for your comments. I should have mentioned that I do use
replication in my cluster, but I'm not sure that the replication is
causing the problem. Another thing to mention about my system is that
there is a lot of data transfer going on most of the time, including
models and data processing applications running on the compute cluster
and data transfers from other sites. I wouldn't be surprised if the
Gluster-NFS handles several terabytes of data before it starts to grind
to a halt. Perhaps this problem hasn't been noticed before because my
usage isn't typical. However, it should be fairly easy to reproduce if
it's just a matter of transferring a large volume of data.
-Dan.
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
