On 02/03/2013 11:22 AM, Michael Colonno wrote:
Having taken a lot more data it does seem the glusterfsd and glusterd
processes (along with several ksoftirqd) spike up to near 100% on both
client and brick servers during any file transport across the mount.
Thankfully this is short-lived for the most part but I'm wondering if
this is expected behavior or what others have experienced(?) I'm a
little surprised such a large CPU load would be required to move small
files and / or use an application within a Gluster mount point.
If you're getting ksoftirqd spikes, that sounds like a hardware issue to
me. I never see huge spikes like that on my servers nor clients.
I wanted to test this against an NFS mount of the same Gluster volume.
I managed to get rstatd installed and running but my attempts to mount
the volume via NFS are met with:
mount.nfs: requested NFS version or transport protocol is not supported
Relevant line in /etc/fstab:
node1:/volume /volume nfs
defaults,_netdev,vers=3,mountproto=tcp 0 0
It looks like CentOS 6.x has NFS version 4 built into everything. So a
few questions:
-Has anyone else noted significant performance differences between a
glusterfs mount and NFS mount for volumes of 8+ bricks?
-Is there a straightforward way to make the newer versions of CentOS
play nice with NFS version 3 + Gluster?
-Are there any general performance tuning guidelines I can follow to
improve CPU performance? I found a few references to the cache
settings but nothing solid.
If the consensus is that NFS will not gain anything then I won't waste
the time setting it all up.
NFS gains you the use of FSCache to cache directories and file stats
making directory listings faster, but it adds overhead decreasing the
overall throughput (from all the reports I've seen).
I would suspect that you have the kernel nfs server running on your
servers. Make sure it's disabled.
Thanks,
~Mike C.
*From:*[email protected]
[mailto:[email protected]] *On Behalf Of *Michael Colonno
*Sent:* Friday, February 01, 2013 4:46 PM
*To:* [email protected]
*Subject:* Re: [Gluster-users] high CPU load on all bricks
Update: after a few hours the CPU usage seems to have dropped down to
a small value. I did not change anything with respect to the
configuration or unmount / stop anything as I wanted to see if this
would persist for a long period of time. Both the client and the
self-mounted bricks are now showing CPU < 1% (as reported by top).
Prior to the larger CPU loads I installed a bunch of software into the
volume (~ 5 GB total). Is this kind a transient behavior -- by which I
mean larger CPU loads after a lot of filesystem activity in short time
-- typical? This is not a problem in my deployment; I just want to
know what to expect in the future and to complete this thread for
future users. If this is expected behavior we can wrap up this thread.
If not then I'll do more digging into the logs on the client and brick
sides.
Thanks,
~Mike C.
*From:*Joe Julian [mailto:[email protected]]
*Sent:* Friday, February 01, 2013 2:08 PM
*To:* Michael Colonno; [email protected]
<mailto:[email protected]>
*Subject:* Re: [Gluster-users] high CPU load on all bricks
Check the client log(s).
Michael Colonno <[email protected] <mailto:[email protected]>>
wrote:
Forgot to mention: on a client system (not a brick) the
glusterfs process is consuming ~ 68% CPU continuously. This is a much
less powerful desktop system so the CPU load can't be compared 1:1
with the systems comprising the bricks but still very high. So the
issue seems to exist with both glusterfsd and glusterfs processes.
Thanks,
~Mike C.
*From:* [email protected]
<mailto:[email protected]>
[mailto:[email protected]] *On Behalf Of *Michael Colonno
*Sent:* Friday, February 01, 2013 12:46 PM
*To:* [email protected] <mailto:[email protected]>
*Subject:* [Gluster-users] high CPU load on all bricks
Gluster gurus ~
I've deployed and 8-brick (2x replicate) Gluster 3.3.1
volume on CentOS 6.3 with tcp transport. I was able to build, start,
mount, and use the volume. On each system contributing a brick,
however, my CPU usage (glusterfsd) is hovering around 20% (virtually
zero memory usage thankfully). These are brand new, fairly beefy
servers so 20% CPU load is quite a bit. The deployment is pretty plain
with each brick mounting the volume to itself via a glusterfs mount. I
assume this type of CPU usage is atypically high; is there anything I
can do to investigate what's soaking up CPU and minimize it? Total
usable volume size is only about 22 TB (about 45 TB total with 2x
replicate).
Thanks,
~Mike C.
------------------------------------------------------------------------
Gluster-users mailing list
[email protected] <mailto:[email protected]>
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
