For completeness:
##############GLUSTER SINGLE DISK NO PERFORMANCE OPTIONS##############
[r...@ac33 gjenos]# time (tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz && sync )
real 0m41.052s
user 0m7.705s
sys 0m3.122s
##############DIRECT SINGLE DISK##############
[r...@ac33 gjenos]# cd /export/jenos
[r...@ac33 jenos]# time (tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz && sync )
real 0m22.093s
user 0m6.932s
sys 0m2.459s
[r...@ac33 jenos]#
The performance options don't appear to be the problem. So the question
stands- how do I get the disk cache advantage through the Gluster
mounted filesystem? It seems to be key in the large performance difference.
Jeremy
On 3/24/2010 4:47 PM, Jeremy Enos wrote:
Good suggestion- I hadn't tried that yet. It brings them much closer.
##############GLUSTER SINGLE DISK##############
[r...@ac33 gjenos]# time (tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz && sync )
real 0m32.089s
user 0m6.516s
sys 0m3.177s
##############DIRECT SINGLE DISK##############
[r...@ac33 gjenos]# cd /export/jenos/
[r...@ac33 jenos]# time (tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz && sync )
real 0m25.089s
user 0m6.850s
sys 0m2.058s
##############DIRECT SINGLE DISK CACHED##############
[r...@ac33 jenos]# time (tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz )
real 0m8.955s
user 0m6.785s
sys 0m1.848s
Oddly, I'm seeing better performance on the gluster system than
previous tests too (used to be ~39 s). The direct disk time is
obviously benefiting from cache. There is still a difference, but it
appears most of the difference disappears w/ the cache advantage
removed. That said- the relative performance issue then still exists
with Gluster. What can be done to make it benefit from cache the same
way direct disk does?
thx-
Jeremy
P.S.
I'll be posting results w/ performance options completely removed from
gluster as soon as I get a chance.
Jeremy
On 3/24/2010 4:23 PM, Bryan Whitehead wrote:
I'd like to see results with this:
time ( tar xzf /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz&&
sync )
I've found local filesystems seem to use cache very heavily. The
untarred file could mostly be sitting in ram with local fs vs going
though fuse (which might do many more sync'ed flushes to disk?).
On Wed, Mar 24, 2010 at 2:25 AM, Jeremy Enos<[email protected]>
wrote:
I also neglected to mention that the underlying filesystem is ext3.
On 3/24/2010 3:44 AM, Jeremy Enos wrote:
I haven't tried all performance options disabled yet- I can try that
tomorrow when the resource frees up. I was actually asking first
before
blindly trying different configuration matrices in case there's a
clear
direction I should take with it. I'll let you know.
Jeremy
On 3/24/2010 2:54 AM, Stephan von Krawczynski wrote:
Hi Jeremy,
have you tried to reproduce with all performance options disabled?
They
are
possibly no good idea on a local system.
What local fs do you use?
--
Regards,
Stephan
On Tue, 23 Mar 2010 19:11:28 -0500
Jeremy Enos<[email protected]> wrote:
Stephan is correct- I primarily did this test to show a demonstrable
overhead example that I'm trying to eliminate. It's pronounced
enough
that it can be seen on a single disk / single node configuration,
which
is good in a way (so anyone can easily repro).
My distributed/clustered solution would be ideal if it were fast
enough
for small block i/o as well as large block- I was hoping that single
node systems would achieve that, hence the single node test.
Because
the single node test performed poorly, I eventually reduced down to
single disk to see if it could still be seen, and it clearly can be.
Perhaps it's something in my configuration? I've pasted my
config files
below.
thx-
Jeremy
######################glusterfsd.vol######################
volume posix
type storage/posix
option directory /export
end-volume
volume locks
type features/locks
subvolumes posix
end-volume
volume disk
type performance/io-threads
option thread-count 4
subvolumes locks
end-volume
volume server-ib
type protocol/server
option transport-type ib-verbs/server
option auth.addr.disk.allow *
subvolumes disk
end-volume
volume server-tcp
type protocol/server
option transport-type tcp/server
option auth.addr.disk.allow *
subvolumes disk
end-volume
######################ghome.vol######################
#-----------IB remotes------------------
volume ghome
type protocol/client
option transport-type ib-verbs/client
# option transport-type tcp/client
option remote-host acfs
option remote-subvolume raid
end-volume
#------------Performance Options-------------------
volume readahead
type performance/read-ahead
option page-count 4 # 2 is default option
option force-atime-update off # default is off
subvolumes ghome
end-volume
volume writebehind
type performance/write-behind
option cache-size 1MB
subvolumes readahead
end-volume
volume cache
type performance/io-cache
option cache-size 1GB
subvolumes writebehind
end-volume
######################END######################
On 3/23/2010 6:02 AM, Stephan von Krawczynski wrote:
On Tue, 23 Mar 2010 02:59:35 -0600 (CST)
"Tejas N. Bhise"<[email protected]> wrote:
Out of curiosity, if you want to do stuff only on one machine,
why do you want to use a distributed, multi node, clustered,
file system ?
Because what he does is a very good way to show the overhead
produced
only by
glusterfs and nothing else (i.e. no network involved).
A pretty relevant test scenario I would say.
--
Regards,
Stephan
Am I missing something here ?
Regards,
Tejas.
----- Original Message -----
From: "Jeremy Enos"<[email protected]>
To: [email protected]
Sent: Tuesday, March 23, 2010 2:07:06 PM GMT +05:30 Chennai,
Kolkata,
Mumbai, New Delhi
Subject: [Gluster-users] gluster local vs local = gluster x4
slower
This test is pretty easy to replicate anywhere- only takes 1
disk, one
machine, one tarball. Untarring to local disk directly vs thru
gluster
is about 4.5x faster. At first I thought this may be due to a
slow
host
(Opteron 2.4ghz). But it's not- same configuration, on a much
faster
machine (dual 3.33ghz Xeon) yields the performance below.
####THIS TEST WAS TO A LOCAL DISK THRU GLUSTER####
[r...@ac33 jenos]# time tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
real 0m41.290s
user 0m14.246s
sys 0m2.957s
####THIS TEST WAS TO A LOCAL DISK (BYPASS GLUSTER)####
[r...@ac33 jenos]# cd /export/jenos/
[r...@ac33 jenos]# time tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
real 0m8.983s
user 0m6.857s
sys 0m1.844s
####THESE ARE TEST FILE DETAILS####
[r...@ac33 jenos]# tar tzvf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz |wc -l
109
[r...@ac33 jenos]# ls -l
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
-rw-r--r-- 1 jenos ac 804385203 2010-02-07 06:32
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
[r...@ac33 jenos]#
These are the relevant performance options I'm using in my .vol
file:
#------------Performance Options-------------------
volume readahead
type performance/read-ahead
option page-count 4 # 2 is default option
option force-atime-update off # default is off
subvolumes ghome
end-volume
volume writebehind
type performance/write-behind
option cache-size 1MB
subvolumes readahead
end-volume
volume cache
type performance/io-cache
option cache-size 1GB
subvolumes writebehind
end-volume
What can I do to improve gluster's performance?
Jeremy
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
