Re: overloaded webserver: nfs wait issue?
N.J. Thomas wrote: * Norberto Meijome [EMAIL PROTECTED] [2005-12-02 11:06:49 +1100]: What's your MaxClients set to? It was set to 256, we actually lowered it to 180. I would have thought you'd want to increase it (after configuring everything else)...else you'll get all those nasty server too busy errors. Running wc -l on the daily Apache access logs, I get: ~1.8million hits per day cool. you should most definitely be able to serve that much (with an http service that is properly configured). Apache should do fine. I dont think i can give much advice on the NFS side of things but in the meantime I would : - increase # of MaxClients (the default is RIDICULOUSLY small, specially in 1.3. You will probably have to recompile with a new max. Higher than 256? yes. IIRC, I've had apache 1.3 configured to over maxclients 1500 with about hardware. Your hardware + OS will set some limits (which you should modify as needed of course), but you should definitely be able to have more than 256. - You RAM seems OK ... you may want to tweak some sysctl or memory settings in Apache (I seem to remember in 1.3 some to do with MMap, but i could be wrong) ... or just add more RAM. Check vmstat (or systat -vm 1) to see how much swapping is going on. Will do...thanks for the suggestions. np btw, i think i forgot to mention you should look into changing to KeepAlive OFF (you want to serve a file and free up the apache resources for the next request - you trade off a bit of speed on each client's transaction, but overall you should see a great improvement. If you dont want to kill http keepalive altogether, set the timeout to a short timeframe (1 minute? 30 secs). good luck, Beto ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
overloaded webserver: nfs wait issue?
We have a website with moderately high traffic, load balanced among 3 webservers. During peak traffic times however (when the volume is higher than normal), the load shoots up to over a 100, and the site crawls to its knees. We set up a script to take snapshots of top every 20 seconds. Here is what it looks like when everthing is normal: 127 last pid: 12003; load averages: 0.93, 1.36, 1.35 up 41+04:22:14 14:00:23 243 processes: 12 running, 230 sleeping, 1 zombie Mem: 222M Active, 74M Inact, 186M Wired, 16M Cache, 111M Buf, 503M Free Swap: 2048M Total, 16M Used, 2032M Free PID USERNAME PRI NICE SIZERES STATETIME WCPUCPU COMMAND 136 root 32 0 1208K 420K RUN 33.1H 7.28% 7.28% amd 11918 nobody-1 0 149M 12292K nfsrcv 0:01 3.00% 1.95% httpd 11879 nobody 2 0 149M 12292K sbwait 0:01 2.10% 1.37% httpd 11896 nobody 2 0 148M 11704K RUN 0:00 1.80% 1.17% httpd 11962 nobody 2 0 147M 10072K RUN 0:00 4.33% 1.12% httpd 11892 nobody-1 0 145M 8804K nfsrcv 0:00 1.35% 0.88% httpd 11935 nobody 2 0 149M 12284K sbwait 0:00 1.73% 0.78% httpd 11925 nobody 2 0 149M 12288K sbwait 0:00 1.08% 0.68% httpd 11894 nobody 2 0 149M 12404K sbwait 0:00 0.98% 0.63% httpd 11937 nobody 2 0 149M 12456K RUN 0:00 1.61% 0.63% httpd 11954 nobody 2 0 149M 12288K sbwait 0:00 1.88% 0.49% httpd 191 root 2 0 144M 6632K select 13:23 0.34% 0.34% httpd 11930 nobody 2 0 145M 8852K sbwait 0:00 0.62% 0.34% httpd 11872 nobody 2 0 149M 12288K sbwait 0:00 0.45% 0.29% httpd 11911 nobody 2 0 148M 11604K accept 0:00 0.45% 0.29% httpd 11893 nobody 2 0 149M 12392K sbwait 0:00 0.38% 0.24% httpd 11876 nobody 2 0 149M 12264K sbwait 0:00 0.38% 0.24% httpd 11934 nobody 2 0 149M 12292K accept 0:00 0.41% 0.20% httpd When the load shoots up, the number of http clients hits Apache's MaxClients setting, here is what top shows: last pid: 12407; load averages: 87.84, 51.91, 27.52 up 41+04:40:51 14:19:00 268 processes: 2 running, 266 sleeping Mem: 715M Active, 68M Inact, 187M Wired, 29M Cache, 111M Buf, 2100K Free Swap: 2048M Total, 272M Used, 1776M Free, 13% Inuse PID USERNAME PRI NICE SIZERES STATETIME WCPUCPU COMMAND 136 root 64 0 1208K 376K RUN 33.1H 2.69% 2.69% amd 11965 nobody-1 0 149M 6892K nfsrcv 0:05 0.24% 0.24% httpd 11913 nobody-1 0 149M 8300K nfsrcv 0:05 0.20% 0.20% httpd 11878 nobody-1 0 149M 8572K nfsrcv 0:09 0.15% 0.15% httpd 11948 nobody-1 0 149M 8852K nfsrcv 0:07 0.15% 0.15% httpd 11982 nobody-1 0 149M 6764K nfsrcv 0:04 0.15% 0.15% httpd 11912 nobody-1 0 149M 4912K nfsrcv 0:06 0.10% 0.10% httpd 12060 nobody-1 0 149M 7356K nfsrcv 0:05 0.10% 0.10% httpd 11999 nobody-1 0 149M 8352K nfsrcv 0:04 0.10% 0.10% httpd 12122 nobody-1 0 149M 8296K nfsrcv 0:04 0.10% 0.10% httpd 12028 nobody-1 0 149M 8664K nfsrcv 0:04 0.10% 0.10% httpd 12267 nobody-1 0 149M 8452K nfsrcv 0:03 0.10% 0.10% httpd 12270 nobody-1 0 150M 7156K nfsrcv 0:02 0.10% 0.10% httpd 11983 nobody-1 0 149M 8256K nfsrcv 0:09 0.05% 0.05% httpd 11977 nobody-1 0 149M 5488K nfsrcv 0:06 0.05% 0.05% httpd 11952 nobody-1 0 149M 6704K nfsrcv 0:06 0.05% 0.05% httpd 11895 nobody-1 0 148M 4404K nfsrcv 0:06 0.05% 0.05% httpd 11885 nobody-1 0 149M 8348K nfsrcv 0:06 0.05% 0.05% httpd The state of all the httpd prcesses are nfsrcv. Does this mean the bottleneck is at the NFS server that hosts the htdocs (and PHP scripts) or just that the server is low on memory? Thomas -- N.J. Thomas [EMAIL PROTECTED] Etiamsi occiderit me, in ipso sperabo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: overloaded webserver: nfs wait issue?
* N.J. Thomas [EMAIL PROTECTED] [2005-12-01 14:38:13 -0500]: We have a website with moderately high traffic, load balanced among 3 webservers. During peak traffic times however (when the volume is higher than normal), the load shoots up to over a 100, and the site crawls to its knees. I forgot to mention that the webservers are running FreeBSD 4.11 and Apache 1.3.x. Thomas -- N.J. Thomas [EMAIL PROTECTED] Etiamsi occiderit me, in ipso sperabo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: overloaded webserver: nfs wait issue?
Can you provide the output of nfsstat -c 1 on the web servers (and nfsstat -s 1 on the nfs server if they are running FreeBSD)? Run them for a minute when you are getting bad performance. Please also quantify moderately high traffic. What is this in Mbit/s? Also, it may be helpful to know a little bit about your network infrastructure, i.e. are the switch ports set to full duplex and the machines to match. NFS performance is very dependent on a fast reliable network. Don't consider anything less than 100Mbit/s full duplex. Also what platform is the NFS server? We recently migrated from a cluster pair of NetApps to a single NFS server running FreeBSD. This is mounted by four NFS clients running Zeus. We have a lot of NFS experience and we felt confident this would work. It did and with the most marginal degradation in performance. Frem. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: overloaded webserver: nfs wait issue?
N.J. Thomas wrote: We have a website with moderately high traffic, load balanced among 3 webservers. During peak traffic times however (when the volume is higher than normal), the load shoots up to over a 100, and the site crawls to its knees. We set up a script to take snapshots of top every 20 seconds. Here is what it looks like when everthing is normal: 127 last pid: 12003; load averages: 0.93, 1.36, 1.35 up 41+04:22:14 14:00:23 243 processes: 12 running, 230 sleeping, 1 zombie Mem: 222M Active, 74M Inact, 186M Wired, 16M Cache, 111M Buf, 503M Free Swap: 2048M Total, 16M Used, 2032M Free [...] When the load shoots up, the number of http clients hits Apache's MaxClients setting, here is what top shows: last pid: 12407; load averages: 87.84, 51.91, 27.52 up 41+04:40:51 14:19:00 268 processes: 2 running, 266 sleeping Mem: 715M Active, 68M Inact, 187M Wired, 29M Cache, 111M Buf, 2100K Free Swap: 2048M Total, 272M Used, 1776M Free, 13% Inuse [...] The state of all the httpd prcesses are nfsrcv. Does this mean the bottleneck is at the NFS server that hosts the htdocs (and PHP scripts) or just that the server is low on memory? Hi Thomas, What's your MaxClients set to? Please define your values for lot of traffic. What CPU? RAM (512MB seems a bit low nowadays)? Disks? I dont think i can give much advice on the NFS side of things but in the meantime I would : - increase # of MaxClients (the default is RIDICULOUSLY small, specially in 1.3. You will probably have to recompile with a new max. - Look at what the PHP scripts do : i.e., is there anything under your control that can be improved? - Install a PHP accelerator (like IonCube's, or Zend's). Or some reverse proxies if you feel like playing with the whole design of your architecture. - if you use mod_gzip, make sure you dont' compress on the fly - it takes a huge hit on the CPU. (Hint: You can pre-compress the files ) - You RAM seems OK ... you may want to tweak some sysctl or memory settings in Apache (I seem to remember in 1.3 some to do with MMap, but i could be wrong) ... or just add more RAM. Check vmstat (or systat -vm 1) to see how much swapping is going on. - Make sure you have tweaked your network sysctl settings (like send + receive buffers, depending on the kind of traffic you get), reduce your close timeouts,etc. - make use of the kernel modules accf_http.ko and accf_data.ko. - disable from Apache ANYTHING that is NOT needed. Enable stuff only as needed on a per virtual host basis. Hope this helps. Beto ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: overloaded webserver: nfs wait issue?
* Norberto Meijome [EMAIL PROTECTED] [2005-12-02 11:06:49 +1100]: What's your MaxClients set to? It was set to 256, we actually lowered it to 180. Please define your values for lot of traffic. Running wc -l on the daily Apache access logs, I get: ~1.8million hits per day What CPU? RAM (512MB seems a bit low nowadays)? Disks? One of the web servers is a (P4, 3GHz, 1GB RAM), the other is a (P3, 1.4GHz, 1GB RAM), they are load balanced evenly, both of them display the same error when the number of httpd processes reach MaxClients. I dont think i can give much advice on the NFS side of things but in the meantime I would : - increase # of MaxClients (the default is RIDICULOUSLY small, specially in 1.3. You will probably have to recompile with a new max. Higher than 256? - Look at what the PHP scripts do : i.e., is there anything under your control that can be improved? I'll ask the devs to take a look. - You RAM seems OK ... you may want to tweak some sysctl or memory settings in Apache (I seem to remember in 1.3 some to do with MMap, but i could be wrong) ... or just add more RAM. Check vmstat (or systat -vm 1) to see how much swapping is going on. Will do...thanks for the suggestions. Thomas -- N.J. Thomas [EMAIL PROTECTED] Etiamsi occiderit me, in ipso sperabo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
