Am 21.01.2010 17:27, schrieb Jiann-Ming Su:
On Wed, Jan 20, 2010 at 5:49 PM, pkoelle<[email protected]> wrote:
This looks suspicious. Is this a gfs2 mount? what kind of performance do you
get wen running bonnie++ directly against /mnt/gluster/gfs2?
Bonnie++ command: ./bonnie++ -u 99:99 -d /mnt/gluster/gfs2 -s 2000
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost.loc 2000M 74 99 39432 12 27308 10 836 99 65104 9 351.5 11
Latency 179ms 1207ms 429ms 109ms 109ms 215ms
Version 1.96 ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 23991 81 +++++ +++ +++++ +++ 17007 57 +++++ +++ +++++ +++
Latency 1276us 1189us 787us 1183us 438us 2813us
Hi Jiann-Ming,
IO for your local disk doesn't look very good either. You are CPU bound
when using putchar() and you get only 74K/sec with 99% CPU. For
reference, I get 68808K/sec with 86% CPU on a XEN vdev on top of cheap
SATA. With a two-node mirror and glusterFS I still get 31886K/sec with
41% CPU (these are the -Per Chr- values from bonnie++).
Can you leave out some of the performance/* translators and also try a
"real" network interface with lower MTU values to see of that helps? I had
terrible results with bigger MTU (6500) and bigger values for
net.core.(r/w)mem_max in /etc/sysctl.conf. IOPS where in the range of 5/sec.
GlusterFS server and client are running on the same system, so there's
no network latency involved at all. I don't see how introducing the
network layer into this would help.
As a matter of fact, with cluster filesystems things get much more
complicated for a single write() or read(). I haven't investigated this
throughoutly but my reasoning is: Data goes through network buffers,
when the buffer is full an interrupt is triggered and data is copied
over. With large buffers this kills small reads and IOPS.
Blame it on gluster or fuse, but
the reality is glusterfs read performance, in a simple base config, is
1/10th of the native file system. In order to have this be worthwhile
at all, I'd have to stripe the data across at least 10 servers to get
the same read performance. This may be a legitimate setup, but it's
not what I need out of gluster. I need the replication capabilities
out of gluster and the ability for the node to read the local copy
reasonably fast.
I can't reproduce. I get reasonable performance given the
hardware,network and all. Do you know any better cluster filesystems? Do
you have experience with gfs2 or ocfs?
cheers
Paul
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
