Somsak Sriprayoonsakul wrote:
I have run html read benchmark test using exactly the same old testset.
- Runing normal httpd (prefork) over NFS-sync option - Average TPS ~ 1500,
Peak TPS ~ 3500
- Running normal httpd (prefork) using booster - Average TPS < 500, Peak TPS
~ 1200, slight improvements but each HTTPD still eat up about 50MB+ (keep
increasing) memory instead of about 20MB without booster. I have to cut the
number of httpd process by half, but still the process used up some swap
space.
- Running httpd.worker using booster - The benchmark result is very low and
error rate is very high. There seems to be some trouble in this mode.
- Here's what I did
- Changed from httpd to httpd.worker in /etc/sysconfig/httpd
- Disable PHP4
- LD_PRELOAD in /etc/init.d/httpd then start httpd
- The HTTPD start correctly and seems to work ok, but not all the time. Two
consecutive wget's on exactly the same URL yield different results
Hi
I can see a few reasons why this could happen with recent
releases. If possible, could you please try the same test
with the release below. In this release, I've fixed a few bugs that
could result in behaviour seen in your tests.
http://ftp.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-2.0.6rc2.tar.gz
Thanks
Shehjar
[r...@compute-0-9 ~]# wget -v --header "Host: www.myhost.local" -O /dev/null
http://c0-3/cafe/siam/topic/F7800428/F7800428.html
--15:52:12-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html
Resolving c0-3... 10.1.255.251
Connecting to c0-3|10.1.255.251|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 264373 (258K) [text/html]
Saving to: `/dev/null'
100%[====================================================================================================================>]
264,373 --.-K/s in 0.003s
15:52:12 (83.3 MB/s) - `/dev/null' saved [264373/264373]
[r...@compute-0-9 ~]# wget -v --header "Host: www.myhost.local" -O /dev/null
http://c0-3/cafe/siam/topic/F7800428/F7800428.html
--15:52:15-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html
Resolving c0-3... 10.1.255.251
Connecting to c0-3|10.1.255.251|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
Saving to: `/dev/null'
[
<=>
] 0 --.-K/s in 0s
15:52:15 (0.00 B/s) - `/dev/null' saved [0/0]
[r...@compute-0-9 ~]#
(Note the returned content-length. Returned contents, when corrected, is
ok)
Here's the error log in booster log file
(First wget)
[2009-08-03 15:36:21] D [libglusterfsclient.c:1340:libgf_vmp_search_entry]
libglusterfsclient: VMP Entry found: /usr/home/www/
www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/
[2009-08-03 15:36:21] D
[libglusterfsclient-dentry.c:381:libgf_client_path_lookup]
libglusterfsclient: resolved path(/www/
www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to
2626532/2650924
[2009-08-03 15:36:21] D [libglusterfsclient.c:1340:libgf_vmp_search_entry]
libglusterfsclient: VMP Entry found: /usr/home/www/
www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/
[2009-08-03 15:36:21] D
[libglusterfsclient-dentry.c:381:libgf_client_path_lookup]
libglusterfsclient: resolved path(/www/
www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to
2626532/2650924
(second wget)
[2009-08-03 15:36:26] D [libglusterfsclient.c:1340:libgf_vmp_search_entry]
libglusterfsclient: VMP Entry found: /usr/home/www/
www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/
[2009-08-03 15:36:26] D
[libglusterfsclient-dentry.c:381:libgf_client_path_lookup]
libglusterfsclient: resolved path(/www/
www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to
2626532/2650924
[2009-08-03 15:36:26] D [libglusterfsclient.c:1340:libgf_vmp_search_entry]
libglusterfsclient: VMP Entry found: /usr/home/www/
www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/
[2009-08-03 15:36:26] D
[libglusterfsclient-dentry.c:381:libgf_client_path_lookup]
libglusterfsclient: resolved path(/www/
www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to
2626532/2650924
In server logs, there are just connect-destroy-connection messages in the
log. Nothing particularily wrong to me.
No such failure occur with Apache+Prefork.
2009/8/2 Somsak Sriprayoonsakul <[email protected]>
After moving the embed configuration file out from glusterfs, now httpd
boot up ok and the web seems to work now.
However, comment posting is not working. It seems that the code that do the
html modification is not working in GlusterFS context. I found that the code
modify web page locally instead. So I think workaround for my case is to
mount glusterfs with fuse at the same path as booster.
I will give it another benchmark and see how it goes.
2009/8/2 Somsak Sriprayoonsakul <[email protected]>
Ok, I have a chance to run booster over 2.0.4
Please find the attach file for my configuration
I did configure boost and try simple ls over my Gluster file system.
Here's the output of ls -al
[r...@compute-0-3 ~]#
LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so ls -l /gluster/www/
ls: /gluster/www/: Invalid argument
ls: /gluster/www/members.pantip.com: Invalid argument
ls: /gluster/www/cafe.pantip.com: Invalid argument
ls: /gluster/www/admin.pantip.com: Invalid argument
ls: /gluster/www/www.pantip.com: Invalid argument
ls: /gluster/www/passwd3.sql: Invalid argument
ls: /gluster/www/passwd.sql: Invalid argument
ls: closing directory /gluster/www/: File descriptor in bad state
total 129972
drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com
drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com
drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com
-rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql
-rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql
drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com
[r...@compute-0-3 ~]#
[r...@compute-0-3 ~]#
LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cp /etc/issue /gluster/
[r...@compute-0-3 ~]#
LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so ls -l /gluster/issue
ls: /gluster/issue: Invalid argument
-rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue
[r...@compute-0-3 ~]#
LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cat /gluster/issue
CentOS release 5.3 (Final)
Kernel \r on an \m
[r...@compute-0-3 ~]#
Despite all those errors, output seems to be fine
And this is what inside my booster.log
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat]
libglusterfsclient: path lookup failed for (/hosts)
[2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat]
libglusterfsclient: path lookup failed for (/issue)
[2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
Then, I try to LD_PRELOAD apache (prefork). I change the target from
/gluster to /usr/home instead (the web application needs it). Then I tried
to strace the httpd process and found that httpd crash at the points where
httpd tried to read configuration file stored on Gluster volume (bad file
descriptor). I will try to move this configuration file some other places
and test again.
2009/7/31 Raghavendra G <[email protected]>
Hi,
On Thu, Jul 30, 2009 at 11:39 AM, Somsak
Sriprayoonsakul<[email protected]> wrote:
Thank you very much for you reply
At the time we used 2.0.3, and yes we used stock Apache from CentOS. I
will
try 2.0.4 very soon to see if it's work.
For Booster, it seems not working correctly for me. Booster complains a
lots
of error with plain 'ls' command (but giving the correct output). Also,
with
Can you mail those errors?
booster, Apache process refuse to start. I will try 2.0.4 to see if it
improves. If not, I will attach error log next time.
logs are very much appreciated.
2009/7/30 Raghavendra G <[email protected]>
Hi Somsak,
Sorry for the delayed reply. Below you've mentioned that you've
problems
with apache and booster. Going forward, Apache over booster will be
the
preferred approach. Can you tell us what version of glusterfs you are
using?
And as I can understand you are using apache 2.2, am I correct?
regards,
----- Original Message -----
From: "Liam Slusser" <[email protected]>
To: "Somsak Sriprayoonsakul" <[email protected]>
Cc: [email protected]
Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat
Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5
performance
issue
I haven't tried an apples to apples comparison with Apache+mod_gluster
vs
Apache+fuse+gluster however i do run both setups. I load tested both
setups
so to verified it could handle 4x our normal daily load and left it at
that.
I didn't actually compare the two (although that might be cool to do
someday).
I really like the idea of Apache+mod_gluster as I don't have to deal
with
the whole fuse and mounting the filesystem. It always scares me
having a
public facing webserver with your whole backend fileshare mounted
locally.
Its very slick for serving content such as media files. We serve
audio
content to our CDN with a pair of Apache/mod_gluster servers - pushing
200-300mbit on average daily and everything works very well.
We run an apache+fuse+gluster setup because we need to run some
mod_perl
before serving the actual content. However performance is still very
good.
We do around 50-100 requests (all jpeg images) per second off of a
fuse
mount and everything works great. We also have a java
tomcat+fuse+gluster
service which does image manipulation on the fly off of a gluster
mount.
We have two backend gluster servers using replication which serve all
this
content.
If you would like more information on our setup id be happy to share
offline. Just email me privately.
thanks,
liam
On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
<[email protected]>wrote:
Oh thank you, thought noone will reply me :)
Have you tried Apache + Fuse over GlusterFS? How is the performance?
Also, anyone in this mailing-list have tried Apache with booster? I
tried
it but Apache refuse to start (just hang and freeze).
2009/7/23 Liam Slusser <[email protected]>
We use mod_gluster and Apache
2.2 with good results. We also ran into the same issue as you that
we
ran out of memory past 150 threads even on a 8gig machine. We got
around
this by compiling Apache using mpm-worker
(threads) instead of prefork - it uses 1/4 as much ram with the
same
number
of connections (150-200) and everything has been running smoothly.
I
cannot
see any performance difference except it using way less memory.
liam
On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
[email protected]> wrote:
Hello,
We have been evaluating the choice for the new platform for a
webboard
system.
The webboard is PHP scripts that generate/modify HTML page when
user
posting/add comment to the page, resulting topic is actually
stored as
a
HTML file with all related file (file attach to the topic, etc..
)stored in
its own directory for each topic. In general, the web site mostly
serve a
lot of small static files using Apache while using PHP to do other
dynamic
contents. This system has been working very well in the past, with
the
increasing page view rate, it is very likely that we will need
some
kind of
Cluster file system as backend very soon.
We have set up a test system using Grinder as stress test tool.
The
test
system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
Apache
(prefork, since the goal is to use this with PHP), linked together
with
Gigabit Ethernet. We try to compare the performance of either
using
single
NFS server in sync mode against using 4 Gluster nodes (distribute
of 2
replicated nodes) through Fuse. However, the transaction per
second
(TPS)
result is not good.
NFS (single server, sync mode)
- 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066,
mean
response time = 61.63 ms
- 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33
ms
- 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
- 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
287.33ms
4 nodes Gluster (2 distribute of replicated 2 node)
- 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =
207.33ms
- 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
672.67ms
- 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt =
931.33
(no 400-600 threads since we run out of client machine, sorry).
gfsd is configured to use 32 thread of iothread as brick.
gfs-client
is
configured to use
io-cache->write-behind->readahead->distribute->replicate.
io-cache cache-size is 256MB. I used patched Fuse downloaded from
Gluster
web-site (build through DKMS).
As the result yield, it seems that Gluster performance worse with
increasing no. of client. One observation is that the glusterfs
process on
client is taking about 100% of CPU during all the tests.
glusterfsd is
utilizing only 70-80% of CPUs during the test time. Note that
system
is Dual
core.
I also tried using modglusterfs and not using fuse at all to serve
all
the static files and conduct another test with Grinder. The result
is
about
the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in
this
test
that each Apache prefork process used more about twice more memory
and
we
need to lower number of httpd processes by about half.
I tried disable EnableMMAP and it didn't help much. Adjusting
readahead,
write behind according to GlusterOptimization page didn't help
much
either.
My question is, there seems to be bottleneck in this setup, but
how
can I
track this? Note that, I didn't do any other optimization other
than
what
said above. Are there any best practice configuration for using
Apache
to
serve a bunch of small static files like this around?
Regards,
Somsak
_______________________________________________
Gluster-users mailing list
[email protected]
http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
regards,
--
Raghavendra G
------------------------------------------------------------------------
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
