Somsak Sriprayoonsakul wrote:
Ok, I have a chance to run booster over 2.0.4
Have you tried configuring booster with the help doc available at:
http://www.gluster.org/docs/index.php/BoosterConfiguration
-Shehjar
Please find the attach file for my configuration
I did configure boost and try simple ls over my Gluster file system. Here's
the output of ls -al
[r...@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
ls -l /gluster/www/
ls: /gluster/www/: Invalid argument
ls: /gluster/www/members.pantip.com: Invalid argument
ls: /gluster/www/cafe.pantip.com: Invalid argument
ls: /gluster/www/admin.pantip.com: Invalid argument
ls: /gluster/www/www.pantip.com: Invalid argument
ls: /gluster/www/passwd3.sql: Invalid argument
ls: /gluster/www/passwd.sql: Invalid argument
ls: closing directory /gluster/www/: File descriptor in bad state
total 129972
drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com
drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com
drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com
-rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql
-rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql
drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com
[r...@compute-0-3 ~]#
[r...@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
cp /etc/issue /gluster/
[r...@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
ls -l /gluster/issue
ls: /gluster/issue: Invalid argument
-rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue
[r...@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
cat /gluster/issue
CentOS release 5.3 (Final)
Kernel \r on an \m
[r...@compute-0-3 ~]#
Despite all those errors, output seems to be fine
And this is what inside my booster.log
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat]
libglusterfsclient: path lookup failed for (/hosts)
[2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat]
libglusterfsclient: path lookup failed for (/issue)
[2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
Then, I try to LD_PRELOAD apache (prefork). I change the target from
/gluster to /usr/home instead (the web application needs it). Then I tried
to strace the httpd process and found that httpd crash at the points where
httpd tried to read configuration file stored on Gluster volume (bad file
descriptor). I will try to move this configuration file some other places
and test again.
2009/7/31 Raghavendra G <[email protected]>
Hi,
On Thu, Jul 30, 2009 at 11:39 AM, Somsak
Sriprayoonsakul<[email protected]> wrote:
Thank you very much for you reply
At the time we used 2.0.3, and yes we used stock Apache from CentOS. I
will
try 2.0.4 very soon to see if it's work.
For Booster, it seems not working correctly for me. Booster complains a
lots
of error with plain 'ls' command (but giving the correct output). Also,
with
Can you mail those errors?
booster, Apache process refuse to start. I will try 2.0.4 to see if it
improves. If not, I will attach error log next time.
logs are very much appreciated.
2009/7/30 Raghavendra G <[email protected]>
Hi Somsak,
Sorry for the delayed reply. Below you've mentioned that you've problems
with apache and booster. Going forward, Apache over booster will be the
preferred approach. Can you tell us what version of glusterfs you are
using?
And as I can understand you are using apache 2.2, am I correct?
regards,
----- Original Message -----
From: "Liam Slusser" <[email protected]>
To: "Somsak Sriprayoonsakul" <[email protected]>
Cc: [email protected]
Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat
Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5
performance
issue
I haven't tried an apples to apples comparison with Apache+mod_gluster
vs
Apache+fuse+gluster however i do run both setups. I load tested both
setups
so to verified it could handle 4x our normal daily load and left it at
that.
I didn't actually compare the two (although that might be cool to do
someday).
I really like the idea of Apache+mod_gluster as I don't have to deal
with
the whole fuse and mounting the filesystem. It always scares me having
a
public facing webserver with your whole backend fileshare mounted
locally.
Its very slick for serving content such as media files. We serve audio
content to our CDN with a pair of Apache/mod_gluster servers - pushing
200-300mbit on average daily and everything works very well.
We run an apache+fuse+gluster setup because we need to run some mod_perl
before serving the actual content. However performance is still very
good.
We do around 50-100 requests (all jpeg images) per second off of a fuse
mount and everything works great. We also have a java
tomcat+fuse+gluster
service which does image manipulation on the fly off of a gluster mount.
We have two backend gluster servers using replication which serve all
this
content.
If you would like more information on our setup id be happy to share
offline. Just email me privately.
thanks,
liam
On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
<[email protected]>wrote:
Oh thank you, thought noone will reply me :)
Have you tried Apache + Fuse over GlusterFS? How is the performance?
Also, anyone in this mailing-list have tried Apache with booster? I
tried
it but Apache refuse to start (just hang and freeze).
2009/7/23 Liam Slusser <[email protected]>
We use mod_gluster and Apache
2.2 with good results. We also ran into the same issue as you that
we
ran out of memory past 150 threads even on a 8gig machine. We got
around
this by compiling Apache using mpm-worker
(threads) instead of prefork - it uses 1/4 as much ram with the same
number
of connections (150-200) and everything has been running smoothly. I
cannot
see any performance difference except it using way less memory.
liam
On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
[email protected]> wrote:
Hello,
We have been evaluating the choice for the new platform for a
webboard
system.
The webboard is PHP scripts that generate/modify HTML page when user
posting/add comment to the page, resulting topic is actually stored
as
a
HTML file with all related file (file attach to the topic, etc..
)stored in
its own directory for each topic. In general, the web site mostly
serve a
lot of small static files using Apache while using PHP to do other
dynamic
contents. This system has been working very well in the past, with
the
increasing page view rate, it is very likely that we will need some
kind of
Cluster file system as backend very soon.
We have set up a test system using Grinder as stress test tool. The
test
system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
Apache
(prefork, since the goal is to use this with PHP), linked together
with
Gigabit Ethernet. We try to compare the performance of either using
single
NFS server in sync mode against using 4 Gluster nodes (distribute of
2
replicated nodes) through Fuse. However, the transaction per second
(TPS)
result is not good.
NFS (single server, sync mode)
- 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
response time = 61.63 ms
- 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33
ms
- 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
- 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
287.33ms
4 nodes Gluster (2 distribute of replicated 2 node)
- 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =
207.33ms
- 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
672.67ms
- 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
(no 400-600 threads since we run out of client machine, sorry).
gfsd is configured to use 32 thread of iothread as brick. gfs-client
is
configured to use
io-cache->write-behind->readahead->distribute->replicate.
io-cache cache-size is 256MB. I used patched Fuse downloaded from
Gluster
web-site (build through DKMS).
As the result yield, it seems that Gluster performance worse with
increasing no. of client. One observation is that the glusterfs
process on
client is taking about 100% of CPU during all the tests. glusterfsd
is
utilizing only 70-80% of CPUs during the test time. Note that system
is Dual
core.
I also tried using modglusterfs and not using fuse at all to serve
all
the static files and conduct another test with Grinder. The result
is
about
the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in
this
test
that each Apache prefork process used more about twice more memory
and
we
need to lower number of httpd processes by about half.
I tried disable EnableMMAP and it didn't help much. Adjusting
readahead,
write behind according to GlusterOptimization page didn't help much
either.
My question is, there seems to be bottleneck in this setup, but how
can I
track this? Note that, I didn't do any other optimization other than
what
said above. Are there any best practice configuration for using
Apache
to
serve a bunch of small static files like this around?
Regards,
Somsak
_______________________________________________
Gluster-users mailing list
[email protected]
http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
regards,
--
Raghavendra G
------------------------------------------------------------------------
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
