On 01/14/2011 06:26 PM, Marcus Bointon wrote:
Our own experience has been generally that you are IOP constrained
because of the stack you have to traverse. If you add more latency
into this stack, you have more to traverse, and therefore, you have
more you need to wait. Which will have a magnification effect upon
times for small IO ops which are seeky (stat, small writes, random
ops).
Sure, and all that applies equally to both NFS and gluster, yet in
Max's example NFS was ~50x faster than gluster for an identical
small-file workload. So what's gluster doing over and above what NFS
is doing that's taking so long, given that network and disk factors
are equal? I'd buy a factor of 2 for replication, but not 50.
If the NFS was doing attribute caching and the GlusterFS implementation
had stat prefetch and other caching turned off, this could explain it.
In case you missed what I'm on about, it was these stats that Max
posted:
Here is the results per command: dd if=/dev/zero of=M/tmp bs=1M
count=16384 69.2 MB/se (Native) 69.2 MB/sec(FUSE) 52 MB/sec (NFS)
dd if=/dev/zero of=M/tmp bs=1K count=163840000 88.1 MB/sec
(Native) 1.1MB/sec (FUSE) 52.4 MB/sec (NFS) time tar cf - M | pv>
/dev/null 15.8 MB/sec (native) 3.48MB/sec (FUSE) 254 Kb/sec (NFS)
Ok, I am not sure if I saw the numbers before. Thanks.
In my case I'm running 30kiops SSDs over gigabit. At the moment my
problem (running 3.0.6) isn't performance but reliability - files are
occasionally reported as 'vanished' by front-end apps (like rsync)
even though they are present on both backing stores; no errors in
gluster logs, self-heal doesn't help.
Check your stat-prefetch settings, and your time base. We've had some
strange issues that seem to be correlated with time bases drifting.
Including files disappearing. We have a few open tickets on this.
The way we've worked around this problem is to abandon the NFS client
and use the glusterfs client. Not our preferred option, but it provides
a workaround for the moment. The NFS translator does appear to have a
few issues. I am hoping we get more tuning knobs for it soon so we can
see if we can work around this.
Regards,
Joe
Marcus _______________________________________________ Gluster-users
mailing list [email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: [email protected]
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
