RE: Performance Problems.. Server hardware smoked by $500 box?
The first step would be to try to quantify the performance difference in serving the actual web pages. Find a single page that you think is slow on the production system and that can be accessed without having to be part of a session, and quantify the performance difference for that page. Remember you don't care about high load, just a single user request. You could use apachebench (ab, comes with apache... something like ab -c 1 -n 20 http://server/path/to/page;) or any simple command line tool that you can time (eg. repeatedly run time GET http://server/path/to/page;). Do this from as near as possible to the box you are running the web browser that sees the slowness loading. I ran a couple of tests, and put the one you requested along with one of the more amusing ones to me, below. These request a page that uses PHP sessions (the sessions are kept over NFS) and a couple of database queries. All 3 servers use the same database, tho obviously the database server handles them locally along with handling the files locally vs over NFS like the other two would. I think to sidestep any more testing that may/may not give us more clues, what we're going to press forward with is trying to get another 133Mhz FSB setup in here since they're so cheap and just testing to see if we see similar results if we set it's harddrive up as a mirror of the production web machine. If by using the same OS installation, and only change the underlying hardware, I'd say that would more-than-likely tell us if this is strictly a hardware issue, assuming it performs equally well as the development machine while using the production machine's setup. John Straiton jks@ clickcom.com Clickcom, Inc 704-365-9970x101 Running ab -c 1 -n 10 on these machines: Production Web over NFS Benchmarking 209.198.22.161 (be patient).done Server Software:Apache/2.0 Server Hostname:209.198.22.161 Server Port:80 Document Length:67769 bytes Concurrency Level: 1 Time taken for tests: 4.918 seconds Complete requests: 10 Failed requests:0 Broken pipe errors: 0 Total transferred: 681350 bytes HTML transferred: 677690 bytes Requests per second:2.03 [#/sec] (mean) Time per request: 491.80 [ms] (mean) Time per request: 491.80 [ms] (mean, across all concurrent requests) Transfer rate: 138.54 [Kbytes/sec] received Production DB serving pages locally Benchmarking 209.198.22.35 (be patient).done Server Software:Apache/1.3.27 Server Hostname:209.198.22.35 Server Port:80 Document Length:68585 bytes Concurrency Level: 1 Time taken for tests: 2.232 seconds Complete requests: 10 Failed requests:0 Broken pipe errors: 0 Total transferred: 689870 bytes HTML transferred: 685850 bytes Requests per second:4.48 [#/sec] (mean) Time per request: 223.20 [ms] (mean) Time per request: 223.20 [ms] (mean, across all concurrent requests) Transfer rate: 309.08 [Kbytes/sec] received Development Server over NFS Benchmarking dev.clickcomworks.net (be patient).done Server Software:Apache/1.3.27 Server Hostname:dev.clickcomworks.net Server Port:8000 Document Length:68237 bytes Concurrency Level: 1 Time taken for tests: 1.799 seconds Complete requests: 10 Failed requests:0 Broken pipe errors: 0 Total transferred: 686380 bytes HTML transferred: 682370 bytes Requests per second:5.56 [#/sec] (mean) Time per request: 179.90 [ms] (mean) Time per request: 179.90 [ms] (mean, across all concurrent requests) Transfer rate: 381.53 [Kbytes/sec] received Now with ab -c 10 -n 10, which is just beautiful Production Web over NFS Benchmarking 209.198.22.161 (be patient).done Server Software:Apache/2.0 Server Hostname:209.198.22.161 Server Port:80 Document Length:67769 bytes Concurrency Level: 100 Time taken for tests: 6.424 seconds Complete requests: 10 Failed requests:0 Broken pipe errors: 0 Total transferred: 817620 bytes HTML transferred: 813228 bytes Requests per second:1.56 [#/sec] (mean) Time per request: 64240.00 [ms] (mean) Time per request: 642.40 [ms] (mean, across all concurrent requests) Transfer rate: 127.28 [Kbytes/sec] received Production DB over NFS Benchmarking 209.198.22.35 (be patient).done Server Software:Apache/1.3.27 Server Hostname:209.198.22.35 Server Port:80 Document Length:68585 bytes Concurrency Level: 10 Time taken for tests: 34.430 seconds Complete requests: 100 Failed requests:0 Broken pipe errors: 0 Total transferred: 6898700 bytes HTML transferred: 6858500 bytes Requests per second:2.90 [#/sec] (mean) Time per request: 3443.00 [ms] (mean) Time per
Performance Problems.. Server hardware smoked by $500 box?
Greets! I'm pretty confused right now with trying to determine the nature of a performance problem I'm having on one of my servers. The server is a webserver with a separate db/file server sitting behind it. The issue is that in pulling up websites from the machine, my silly POS development box has nearly double performance although one would think it shouldn't. Serving pages from the production machine over NFS or even the db/file server via local filesystem is slower than pages from the development machine over NFS. By that I mean that if you view the page on the development machine over the LAN, it appears instantly...Do that on the production machines and 1-2 seconds pass as you watch images flow in... With zero load. Now granted the development machine has the most Mhz, but we're talking *visible* differences in speed between a 1.5GhzAMD ~1Ghz Intel Server Class machines. And to top it all off, with the miniscule amount of ram in the AMD box, it's paging around 50MB (with only a few K free of physical ram) by the time apache finshes spawning during the boot. By all rights, to me, this thing should be no better than equivalent to the production machines. They are all plugged in to the same Cisco 3524 switch, and I've tried mulitple cables (even swapping between the fast slow machines to try to hurt the fast one) and the IOS reports no errors on the interfaces, all at 100/FullDuplex. I can reproduce the problem on any # of viewing machines at will. Please tell me that I shouldn't toss these multi-thousand dollar servers and just buy a bunch of these silly do-it-yourself'ers..(*kidding*, but you get the idea) The hardware involved: Production Machine: (Dell PowerEdge 1U server 1G/UWSCSI) FreeBSD 5.1-Current (Was 4.8 Release, upgraded in attempt to find problem) CPU: Intel Pentium III (927.11-MHz 686-class CPU) Origin = GenuineIntel real memory = 1342111744 (1279 MB) fxp0: Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet da0: SEAGATE ST318305LC 2203 Fixed Direct Access SCSI-3 device da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit) da0: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C) DB/File Server: (Dell PowerEdge 2U Server 768MB/RAID5) === FreeBSD 4.8 Stable CPU: Intel(R) Pentium(R) III CPU family 1266MHz (1263.45-MHz 686-class CPU) real memory = 805240832 (786368K bytes) fxp0: Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet aac0: Dell PERC 3/Di mem 0xf000-0xf7ff irq 2 at device 2.1 on pci2 aac0: i960RX 100MHz, 118MB cache memory, optional battery present aacd0: RAID 5 on aac0 aacd0: 69425MB (142182912 sectors) Development Machine: (Whitebox machine) FreeBSD 5.0-Release CPU: AMD Athlon(tm) XP 1800+ (1544.67-MHz 686-class CPU) real memory = 234815488 (223 MB) rl0: RealTek 8139 10/100BaseTX ad0: 39205MB Maxtor 6E040L0 [79656/16/63] at ata0-master UDMA133 Ideas? Could the CPU alone be doing that much towards the difference? RAM (DDRvsECC)? I don't have any other AMD machines to use in a control group so I can't persue that route. I have benchmarks attached below... Thanks for any ideas, John Straiton jks@ clickcom.com Clickcom, Inc 704-365-9970x101 The benchmarks that show the most difference are these...my notes in ()'s: #bonnie -s 1(run on NFS share) ---Sequential Output ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- MachineMB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU PROD1 6364 25.0 9121 4.9 166 0.3 2906 13.5 94108 28.6 398.8 3.3 ---Sequential Output ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- MachineMB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU DEV 1 11246 65.3 11259 6.4 6733 4.3 11009 41.9 292571 95.3 5612.2 28.9 #ubench PROD: (ECC SDRAM) Ubench CPU:25713 Ubench MEM:22707 Ubench AVG:24210 DEV: (DDR) Ubench CPU:66048 Ubench MEM:57434 Ubench AVG:61741 #netperf (TO DB MACHINE FROM NOTED MACHINE) TCP STREAM TEST to DB FROM DEV Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 57344 32768 3276810.01 92.74 TCP STREAM TEST to DB FROM PROD Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 57344 32768 3276810.00 87.61 ___ [EMAIL PROTECTED] mailing list
RE: Performance Problems.. Server hardware smoked by $500 box?
I'm pretty confused right now with trying to determine the nature of a performance problem I'm having on one of my servers. The server is a webserver with a separate db/file server sitting behind it. The issue is that in pulling up websites from the machine, my silly POS development box has nearly double performance although one would think it shouldn't. [...] Disclaimer: these are random, uninformed guesses... - Is it possible the server has too much RAM? I'll look into this. - Are you using the same db server as the backend for your development box? Thanks for your suggestions, unfortunately- yes, both scenerios in production (webserver + db server or db server acting as both) are slower than the development box serving off of the db server. There are no local services other than apache on the development machine. John Straiton jks@ clickcom.com Clickcom, Inc 704-365-9970x101 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Performance Problems.. Server hardware smoked by $500 box?
On Thu, 11 Sep 2003, John Straiton wrote: Greets! I'm pretty confused right now with trying to determine the nature of a performance problem I'm having on one of my servers. The server is a webserver with a separate db/file server sitting behind it. The issue is that in pulling up websites from the machine, my silly POS development box has nearly double performance although one would think it shouldn't. you need to quantify the performance differences starting with replicating what you see, and work down from there. Do the differences in the numbers you posted mean anything? No idea. I wouldn't expect them to make any difference for a single user hitting the server, but it is possible they are related to some problem somewhere. The easiest way to tell, however, is to start with the problem you do see (ie. things are slow loading) and work down through the software stack from there. The first step would be to try to quantify the performance difference in serving the actual web pages. Find a single page that you think is slow on the production system and that can be accessed without having to be part of a session, and quantify the performance difference for that page. Remember you don't care about high load, just a single user request. You could use apachebench (ab, comes with apache... something like ab -c 1 -n 20 http://server/path/to/page;) or any simple command line tool that you can time (eg. repeatedly run time GET http://server/path/to/page;). Do this from as near as possible to the box you are running the web browser that sees the slowness loading. Until you can reproduce and quantify a performance difference at this level, don't worry about digging deeper. Once you can, keep taking one step closer. Try requesting a page that doesn't hit the database. Try setting things up so NFS isn't being used. Try making the request from the same machine the web server is running on. etc. Your goal here is to eliminate as many components as possible while still being able to reproduce the high level problem. So, for example, if you can reproduce it on a page that doesn't hit the database... you can eliminate that from further consideration. Unless your application is extremely heavyweight and demanding on hardware, or there is some bug in one of the drivers or configuration, none of the hardware differences would normally have any effect on the symptoms you say you are seeing. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Performance Problems.. Server hardware smoked by $500 box?
John Straiton writes: I'm pretty confused right now with trying to determine the nature of a performance problem ... on one of my servers. ... in pulling up websites from the machine, my silly POS development box has nearly double performance ... There's lots of tricky stuff that can be going wrong. I spent some time in my last two jobs (anybody got a new one in NJ?) on speeding up stuff like this and the first thing I try to do is put some kind of steady-state load on the boxen and monitor each box involved with systat 1 -vmstat . There's one hell of a lot of information there, and interactions are sometimes hard to see. If the CPU is fully occupied, it could be the network stack (which will NOT show up at interrupt level) and that can depend on what interface chipset you're using as well. Or it could be ... well, get the data first. If you'd like to send me a few sample screens, I'll try to make suggestions on what to check next. You want to have a series from each of the three configurations you're using. And being able to _watch_ what's happening on systat is worth a whole lot of non-sequenced snapshots. Are you running firewall software on the production machine? I don't know how the FreeBSD version will affect performance, but it can't help. How about the reports from top ? What do they say? What's soaking up the processor? Running on a 1GHz PIII two years ago, I was able to get a web proxy (not squid!) to serve 1500+ requests per second, with about 200 MBit/sec of ethernet traffic (inbound and out). (The product never made it into full-scale production, largely due to financial problems in the large, well-known corporation.) So the problem isn't horsepower, but something not using it well. Can you try running the back end box on a simple disk without the RAID in the way? I don't recall all the properties of RAID 5 right now, but in general RAID trades disk transactions away to get disk throughput. In your application, you probably need transactions more than throughput. Dumb question: have you tried swapping cables/ports on the ethernet connections? Does one link support jumbo frames and the other not? How about network buffers: have you got enough configured, and how many are tied up at a time? Performance is often a negative art: find the worst roadblock and remove it, then the next worst after that, and so forth. Mark Terribile __ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Performance Problems.. Server hardware smoked by $500 box?
Post your kernel configs, or better yet, do a diff -u between the 5.0-R and the 5.1-C kernel configs. I bet dime to dollar you've got some debugging options enabled in the 5.1-C config. At the very least you haven't remove the debugging options from your malloc options. *frown* The 5.0 development machine is running GENERIC. Being that we never customized the kernel, we don't have a backup copy of it. I can say since I'd have been the one to change it, that the only differences between GENERIC and what we put into service is that we always remove the device listings for things we don't have any use for, like ISA nic's and such. I almost never change any of the other features. If 5.1-C has debugging on by default then , yes, I'd concur that we have those features turned on. However the production machine was 4.8-R when we noticed the problem. From what it sounds, it should have been faster due to that fact. The only thing that's been constant in this situation is that the development machine hasn't changed. Everything I've done has been trying to change variables on the production machine to either match or surpass the development to bring it up to it's speed. Roadmap for the production machine so far: Upgrade apache/php to newest in ports. Add RAM Upgrade OS to 5.1 from 4.8, reinstall every package in pkg_info Update apache to 2.X from 1.3.X reinstall php as is required Swap network interfaces between the two onboard ones Swap ethernet cables with the development machine Swap ethernet ports with the development machine John Straiton jks@ clickcom.com Clickcom, Inc 704-365-9970x101 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Performance Problems.. Server hardware smoked by $500 box?
There's lots of tricky stuff that can be going wrong. I spent some time in my last two jobs (anybody got a new one in NJ?) on speeding up stuff like this and the first thing I try to do is put some kind of steady-state load on the boxen and monitor each box involved with systat 1 -vmstat . There's one hell of a lot of information there, and interactions are sometimes hard to see. If the CPU is fully occupied, it could be the network stack (which will NOT show up at interrupt level) and that can depend on what interface chipset you're using as well. Or it could be ... well, get the data first. If you'd like to send me a few sample screens, I'll try to make suggestions on what to check next. You want to have a series from each of the three configurations you're using. And being able to _watch_ what's happening on systat is worth a whole lot of non-sequenced snapshots. I'll take a look at this. Are you running firewall software on the production machine? Are you trying to hack me? *chuckle* j/k I don't know how the FreeBSD version will affect performance, but it can't help. How about the reports from top ? What do they say? What's soaking up the processor? If you saw the two machines, rc.d/* and rc.conf are nearly identical. There aren't any services that run on one that aren't exactly the same on the other except the differences between the OS versions as best I can tell. In looking at top -qSi (or even normal) the only things that tend to show up at the top are httpd processes. The development machine has the ata irq up there whereas the production doesn't seem to have disk access in the top few entries. Just reinforcing that even with more to do, the development machine smokes this Dell iron. Can you try running the back end box on a simple disk without the RAID in the way? I don't recall all the properties of RAID 5 right now, but in general RAID trades disk transactions away to get disk throughput. In your application, you probably need transactions more than throughput. No, it has no other interfaces configured that I could get into without taking the machine down to hook up some cables. This thing boots of the hardware RAID and has a cage in the front for the drives (hotswap). But being that the RAID is one of the few things that IS the same between production and development (the raid serves files via NFS to both incarnations of the webserver in the same fashion) , I'd have to get an idea why this was something I'd want to delve further into. Dumb quesion: have you tried swapping cables/ports on the ethernet connections? Does one link support jumbo frames and the other not? How about network buffers: have you got enough configured, and how many are tied up at a time? I'm not familiar with those configuration options but if it makes a difference, I basically use completely default setups between the xl0 and fxp0 drivers being used in these scenarios, with GENERIC being the kernel for all intents and purposes between both machines. Unless one driver is set up different than the other or has inheirent speed advantages, they should both be equivalent. Yes, all cabling/ports and even in the case of the production machine, interfaces have been swapped/changed. Another good idea that didn't pan out... Thanks for the help! John Straiton jks@ clickcom.com Clickcom, Inc 704-365-9970x101 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Performance Problems.. Server hardware smoked by $500 box?
Post your kernel configs, or better yet, do a diff -u between the 5.0-R and the 5.1-C kernel configs. I bet dime to dollar you've got some debugging options enabled in the 5.1-C config. At the very least you haven't remove the debugging options from your malloc options. *frown* The 5.0 development machine is running GENERIC. Being that we never customized the kernel, we don't have a backup copy of it. I can say since I'd have been the one to change it, that the only differences between GENERIC and what we put into service is that we always remove the device listings for things we don't have any use for, like ISA nic's and such. I almost never change any of the other features. Removing unused drivers shouldn't make any difference. If 5.1-C has debugging on by default then , yes, I'd concur that we have those features turned on. However the production machine was 4.8-R when we noticed the problem. From what it sounds, it should have been faster due to that fact. The only thing that's been constant in this situation is that the development machine hasn't changed. Everything I've done has been trying to change variables on the production machine to either match or surpass the development to bring it up to it's speed. hrm... On your development box, do you have any boot time configuration options set, how about differing values in sysctl.conf? Hrm, let me re-read your original post. Apache: Are you sure you have your reverse DNS setup correctly? Are you doing host name resolution in Apache for your logs? If so, turn that off! bonnie: It's very possible that the CPU is making a difference here if bonnie is getting near 100% hit rates for the cache. Given that you're getting good throughput, I'd double check that you're machines are at 100Mbps-TX, full duplex in ifconfig. I bet one of them is half-duplex and that's the difference there. I'm betting there are some sysctl's that are different between these machines, along with possibly Apache and your DNS not being setup correctly. Roadmap for the production machine so far: Upgrade apache/php to newest in ports. I haven't seen anyone claim that Apache2 is faster than Apache 1. If Apache2 is faster, could someone provide some evidence? Add RAM This won't make a difference unless you're swapping our out of RAM. Upgrade OS to 5.1 from 4.8, reinstall every package in pkg_info Downgrade you mean? :) 4.8 is going to be more battle proven than 5.1, so I'd recommend using 4.8 for that reason alone. If you do want to use 5.1 for the sake of helping 5.1 become more mature, very cool. Update apache to 2.X from 1.3.X reinstall php as is required See above comment on Apache. Swap network interfaces between the two onboard ones This could make some difference, but 6MBps is more than plenty throughput for webserving via NFS. Swap ethernet cables with the development machine This would impact network performance, something else is going on. Swap ethernet ports with the development machine Again, this would impact network performance, and at 6MBps, that's well more than enough for you to get reasonable performance out of a 5.1 or 4.8 box. Something basic and simple is going on here. DNS: dig my.test.box.example.com. a dig d.c.b.a.in-addr.arpa. ptr apache: grep Hostname /usr/local/etc/apache/httpd.conf -sc -- Sean Chittenden ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Performance Problems.. Server hardware smoked by $500 box?
On Thu, Sep 11, 2003 at 05:30:59PM -0400, John Straiton wrote: If 5.1-C has debugging on by default then , yes, I'd concur that we have those features turned on. 5.1-CURRENT indeed has a number of debugging features enabled by default, which can cause significant performance loss under load. Kris pgp0.pgp Description: PGP signature
