I wrote up a text file about benchmarking and comparing Tomcat-4.0-M5 (pre-release) and Apache 1.3.12. It's attached to this message. I wrote it for anyone who is interested (even non-Java-saavy people) to know how the raw content serving performance of Catalina and its built-in web server compares to that of Apache 1.3.12. It contains lots of information to help people understand some of the important differences between the two servers. Feedback, flames, reproduced test results, etc. are welcome! :) Cheers. -- Jason Brittain Software Engineer, Olliance Inc. http://www.Olliance.com Current Maintainer, Locomotive Project http://www.Locomotive.org
I've just completed some new benchmarks comparing the static content serving performance of Tomcat/Catalina M5 (pre-release from CVS) and Apache 1.3.12. Below I've included information about the benchmark, and the results. About My Apache Server 1.3.12 I downloaded Apache 1.3.12 source release from http://www.apache.org/ , compiled it with gcc, and installed it on my machine. Here is the module signature of my apache httpd: [jason@netninja bin]$ ./httpd -l Compiled-in modules: http_core.c mod_env.c mod_log_config.c mod_mime.c mod_negotiation.c mod_status.c mod_include.c mod_autoindex.c mod_dir.c mod_cgi.c mod_asis.c mod_imap.c mod_actions.c mod_userdir.c mod_alias.c mod_access.c mod_auth.c mod_so.c mod_setenvif.c suexec: disabled; invalid wrapper /usr/local/apache/bin/suexec Here is a description of the only configuration options I changed (in conf/httpd.conf): Configuration Variable Old Value New Value ------------------------------------------------------------------ MaxKeepAliveRequests 100 200 MinSpareServers 5 50 MaxSpareServers 10 250 StartServers 5 50 MaxClients 150 250 I set MaxKeepAliveRequests higher so that the server would not run into a ceiling with keepalive requests (for better performance). I set the SpareServers variables and StartServer to higher numbers so that the server would not have to for lots of new servers at the beginning of the test (which would probably severely degrade the performance -- this is one major scalability problem with Apache 1.3 that Apache 2.0 aims to solve). I also raised MaxClients above the level that it should need to be for my tests, removing another potential ceiling. With this configuration, after running the server, ps shows that Apache has 53 httpd processes running and ready to serve. About Tomcat 4.0 / Catalina M5 I pulled the Tomcat 4.0 M5 pre-release sources from cvs (see http://jakarta.apache.org for information about the source modules. The code was the HEAD revision from CVS available on November 30th), built the source using Sun's JDK 1.3, and installed the package on my machine. The only settings that I changed are in the server.xml file: In the config section that defines the Catalina stand-alone web server's threading settings, here's what the original config said: <!-- Define a non-SSL HTTP/1.1 Connector on port 8080 --> <Connector className="org.apache.catalina.connector.http.HttpConnector" port="8080" minProcessors="5" maxProcessors="75" acceptCount="10" debug="0"/> .. and here's what I changed it to: <!-- Define a non-SSL HTTP/1.1 Connector on port 8080 --> <Connector className="org.apache.catalina.connector.http.HttpConnector" port="8080" minProcessors="100" maxProcessors="210" acceptCount="255" debug="0"/> Plus, I turned off some timestamping of other log files. Here's the diff between the stock server.xml file and my server.xml file: 47,48c47,48 < port="8080" minProcessors="5" maxProcessors="75" < acceptCount="10" debug="0"/> --- > port="8080" minProcessors="100" maxProcessors="210" > acceptCount="255" debug="0"/> 78c78 < timestamp="true"/> --- > timestamp="false"/> 122c122 < timestamp="true"/> --- > timestamp="false"/> 139c139 < timestamp="true"/> --- > timestamp="false"/> Then, I set the CATALINA_OPTS (optional settings to be passed to the JVM) environment variable to a value of "-Xms96m -Xmx96m". What this does is give the server a startup heap of 96M of RAM (all it will need so that it does not need to resize) and a maximum heap size of 96M. This single setting greatly inproved Catalina's performance compared to the same tests without this setting! The Java VM I used for running the test is Sun's J2SE 1.3 HotSpot Client VM for Linux: [jason@netninja conf]$ java -version java version "1.3.0beta_refresh" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0beta_refresh-b09) Java HotSpot(TM) Client VM (build 1.3.0beta-b07, mixed mode) Last, I copied the Apache index.html file into the document root for Catalina, so that the test serves the exact same content on both servers. My Machine It's a Dell Inspiron 5000 notebook with a 500Mhz Pentium III Coppermine processor, 256M of RAM. Here is my processor information: [jason@netninja conf]$ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 8 model name : Pentium III (Coppermine) stepping : 1 cpu MHz : 497.839198 cache size : 128 KB fdiv_bug : no hlt_bug : no sep_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr xmm bogomips : 496.44 Here is information about my hard drive bus, from dmesg: PIIX4: IDE controller on PCI bus 00 dev 39 PIIX4: not 100% native mode: will probe irqs later ide0: BM-DMA at 0x1050-0x1057, BIOS settings: hda:DMA, hdb:pio ide1: BM-DMA at 0x1058-0x105f, BIOS settings: hdc:DMA, hdd:pio hda: IBM-DARA-212000, ATA DISK drive hdc: TOSHIBA CD-ROM XM-7002B, ATAPI CDROM drive hda: IBM-DARA-212000, 11513MB w/418kB Cache, CHS=1559/240/63 hdc: ATAPI 24X CD-ROM drive, 128kB Cache My machine runs RedHat Linux 6.2, with a kernel that I compiled to support some devices on this machine: [jason@netninja conf]$ uname -a Linux netninja.olliance.com 2.2.14-12 #1 Tue Apr 25 13:04:07 EDT 2000 i686 unknown The Test I wanted to benchmark these servers in a way that is similar to real-world web serving, so I chose to benchmark concurrent requests from many clients at once. The client tester tool that I used is ApacheBench ("ab"). It came with the Apache that was pre-installed on my RedHat 6.2 system: [root@netninja /root]# which ab /usr/sbin/ab [root@netninja /root]# ab -h Usage: ab [options] [http://]hostname[:port]/path Options are: -n requests Number of requests to perform -c concurrency Number of multiple requests to make -t timelimit Seconds to max. wait for responses -p postfile File containg data to POST -T content-type Content-type header for POSTing -v verbosity How much troubleshooting info to print -w Print out results in HTML tables -i Use HEAD instead of GET -x attributes String to insert as table attributes -y attributes String to insert as tr attributes -z attributes String to insert as td or th attributes -C attribute Add cookie, eg. 'Apache=1234. (repeatable) -H attribute Add Arbitrary header line, eg. 'Accept-Encoding: zop' Inserted after all normal header lines. (repeatable) -A attribute Add Basic WWW Authentication, the attributes are a colon separated username and password. -p attribute Add Basic Proxy Authentication, the attributes are a colon separated username and password. -V Print version number and exit -k Use HTTP KeepAlive feature -h Display usage information (this message) And, info from the manual page for ab: ab(1) ab(1) NAME ab - Apache HTTP server benchmarking tool SYNOPSIS ab [ -k ] [ -i ] [ -n requests ] [ -t timelimit ] [ -c concurrency ] [ -p POST file ] [ -A Authenticate user name:password ] [ -P Proxy Authenticate username:password ] [ -H Custom header ] [ -C Cookie name=value ] [ -T con tent-type ] [ -v verbosity ] ] [ -w output HTML ] ] [ -x <table> attributes ] ] [ -y <tr> attributes ] ] [ -z <td> attributes ] [http://]hostname[:port]/path ab [ -V ] [ -h ] DESCRIPTION ab is a tool for benchmarking the performance of your Apache HyperText Transfer Protocol (HTTP) server. It does this by giving you an indication of how many requests per second your Apache installation can serve. For my test, I used ab to simulate 60 concurrent clients that make a total of 10,000 requests to the server, completing them as fast as the server is capable of completing them. At the end of each benchmark, ab prints the timed results. I used two sets of switches to ab: one that uses HTTP keepalive connections, and another that doesn't: /usr/sbin/ab -n 10000 -c 60 http://localhost:8080/index.html (no keepalives) /usr/sbin/ab -k -n 10000 -c 60 http://localhost:8080/index.html (keepalives on) Note that these switches tell ab to run 60 concurrent clients, each making requests to the server concurrently, for an overall total number of requests being 10,000. During the tests, I ran either Apache, or Catalina (not both), and I allowed each test to complete without running or shutting down any other processes. Here is output (and benchmark numbers) from the tests: ========= Apache 1.3 Benchmarks =================== [root@netninja /root]# /usr/sbin/ab -k -n 10000 -c 60 http://localhost:80/index.html This is ApacheBench, Version 1.3c <$Revision: 1.38 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-1999 The Apache Group, http://www.apache.org/ Benchmarking localhost (be patient)... Server Software: Apache/1.3.12 Server Hostname: localhost Server Port: 80 Document Path: /index.html Document Length: 1358 bytes Concurrency Level: 60 Time taken for tests: 23.978 seconds Complete requests: 10000 Failed requests: 0 Keep-Alive requests: 10043 Total transferred: 18364664 bytes HTML transferred: 13638394 bytes Requests per second: 417.05 Transfer rate: 765.90 kb/s received Connnection Times (ms) min avg max Connect: 0 2 645 Processing: 112 140 7074 Total: 112 142 7719 [root@netninja /root]# /usr/sbin/ab -k -n 10000 -c 60 http://localhost:80/index.html This is ApacheBench, Version 1.3c <$Revision: 1.38 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-1999 The Apache Group, http://www.apache.org/ Benchmarking localhost (be patient)... Server Software: Apache/1.3.12 Server Hostname: localhost Server Port: 80 Document Path: /index.html Document Length: 1358 bytes Concurrency Level: 60 Time taken for tests: 23.353 seconds Complete requests: 10000 Failed requests: 0 Keep-Alive requests: 10020 Total transferred: 18322620 bytes HTML transferred: 13607160 bytes Requests per second: 428.21 Transfer rate: 784.59 kb/s received Connnection Times (ms) min avg max Connect: 0 1 365 Processing: 134 137 139 Total: 134 138 504 [root@netninja /root]# /usr/sbin/ab -n 10000 -c 60 http://localhost:80/index.html This is ApacheBench, Version 1.3c <$Revision: 1.38 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-1999 The Apache Group, http://www.apache.org/ Benchmarking localhost (be patient)... Server Software: Apache/1.3.12 Server Hostname: localhost Server Port: 80 Document Path: /index.html Document Length: 1358 bytes Concurrency Level: 60 Time taken for tests: 35.188 seconds Complete requests: 10000 Failed requests: 0 Total transferred: 17945820 bytes HTML transferred: 13607160 bytes Requests per second: 284.19 Transfer rate: 510.00 kb/s received Connnection Times (ms) min avg max Connect: 71 102 476 Processing: 133 107 148 Total: 204 209 624 [root@netninja /root]# /usr/sbin/ab -n 10000 -c 60 http://localhost:80/index.html This is ApacheBench, Version 1.3c <$Revision: 1.38 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-1999 The Apache Group, http://www.apache.org/ Benchmarking localhost (be patient)... Server Software: Apache/1.3.12 Server Hostname: localhost Server Port: 80 Document Path: /index.html Document Length: 1358 bytes Concurrency Level: 60 Time taken for tests: 34.890 seconds Complete requests: 10000 Failed requests: 0 Total transferred: 17945820 bytes HTML transferred: 13607160 bytes Requests per second: 286.62 Transfer rate: 514.35 kb/s received Connnection Times (ms) min avg max Connect: 71 102 176 Processing: 133 105 97 Total: 204 207 273 ========= Catalina Benchmarks ===================== [jason@netninja jason]$ /usr/sbin/ab -k -n 10000 -c 60 http://localhost:8080/index.html This is ApacheBench, Version 1.3c <$Revision: 1.38 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-1999 The Apache Group, http://www.apache.org/ Benchmarking localhost (be patient)... Server Software: Apache Server Hostname: localhost Server Port: 8080 Document Path: /index.html Document Length: 1358 bytes Concurrency Level: 60 Time taken for tests: 46.361 seconds Complete requests: 10000 Failed requests: 0 Keep-Alive requests: 0 Total transferred: 15869538 bytes HTML transferred: 13588148 bytes Requests per second: 215.70 Transfer rate: 342.30 kb/s received Connnection Times (ms) min avg max Connect: 0 80 491 Processing: 19 197 371 Total: 19 277 862 [jason@netninja jason]$ /usr/sbin/ab -k -n 10000 -c 60 http://localhost:8080/index.html This is ApacheBench, Version 1.3c <$Revision: 1.38 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-1999 The Apache Group, http://www.apache.org/ Benchmarking localhost (be patient)... Server Software: Apache Server Hostname: localhost Server Port: 8080 Document Path: /index.html Document Length: 1358 bytes Concurrency Level: 60 Time taken for tests: 44.273 seconds Complete requests: 10000 Failed requests: 0 Keep-Alive requests: 0 Total transferred: 15880808 bytes HTML transferred: 13597654 bytes Requests per second: 225.87 Transfer rate: 358.70 kb/s received Connnection Times (ms) min avg max Connect: 0 102 268 Processing: 38 162 324 Total: 38 264 592 [jason@netninja jason]$ /usr/sbin/ab -n 10000 -c 60 http://localhost:8080/index.html This is ApacheBench, Version 1.3c <$Revision: 1.38 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-1999 The Apache Group, http://www.apache.org/ Benchmarking localhost (be patient)... Server Software: Apache Server Hostname: localhost Server Port: 8080 Document Path: /index.html Document Length: 3182 bytes Concurrency Level: 60 Time taken for tests: 43.665 seconds Complete requests: 10000 Failed requests: 0 Total transferred: 34168200 bytes HTML transferred: 31883640 bytes Requests per second: 229.02 Transfer rate: 782.51 kb/s received Connnection Times (ms) min avg max Connect: 2 126 271 Processing: 240 134 32 Total: 242 260 303 [jason@netninja jason]$ /usr/sbin/ab -n 10000 -c 60 http://localhost:8080/index.html This is ApacheBench, Version 1.3c <$Revision: 1.38 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-1999 The Apache Group, http://www.apache.org/ Benchmarking localhost (be patient)... Server Software: Apache Server Hostname: localhost Server Port: 8080 Document Path: /index.html Document Length: 3182 bytes Concurrency Level: 60 Time taken for tests: 43.818 seconds Complete requests: 10000 Failed requests: 0 Total transferred: 34168200 bytes HTML transferred: 31883640 bytes Requests per second: 228.22 Transfer rate: 779.78 kb/s received Connnection Times (ms) min avg max Connect: 2 122 262 Processing: 230 139 39 Total: 232 261 301 ================================================= Other Notes About The Tests In my tests, the client (ApacheBench) and the server were both run on the same machine (I needed it to be mobile). That means that the system running the server is also busy due to the client's resource needs. I would like to re-run these tests as two separate machines: one for the client and one for the server. That way, the numbers can more closely indicate the performance of the servers on the machine the test was run on. Running the client on the same machine as the server shouldn't be unfair to either server, since the client is the same in both tests. But, to be sure, it's best to eliminate the client and isolate just the server. Also, all tests ran over local loopback network connections, not over ethernet. Something about ab that I noticed was that it does not implement a true HTTP 1.1 client -- only an HTTP 1.0 client that does 1.0 keepalive connections. Also, ab's numbers didn't always make sense. Sometimes the maximum times were smaller than the minimum times.. Sometimes the average times were smaller than the minimum times. I don't have an explanation (other than this is a bug in ab), but in general it did seem to give useful benchmark information. In the benchmark run output of ab, it always says "Server Software: Apache", even if the server isn't Apache. This is another bug in ab. For the Catalina benchmarks, the first test after a fresh run of the server will always be slow, since the Java VM needs a small amount of running time to utilize the VM's runtime optimization features. It natively compiles classes and methods that are deemed to be the busy spots of the code (or the "HotSpots"), and after they're natively compiled the server runs faster. The longer the server runs, the faster it runs, until the point where most everything is natively compiled (at which time, everything runs its fastest). I made no attempt here to compare the memory footprints of these two servers. It may not be easy to determine how much memory Apache uses for its pool of 50 or more httpd processes (it's dynamic). Nowadays, with the low price of RAM, it's usually not a problem to give the servers the memory they need. But, in the case where you are low on RAM, a good comparison between the two memory footprints (and the characteristics of dynamic memory usage choices each offers) is a good idea. Finally, both servers could probably be further optimized for speed. For instance, Apache could be compiled with gcc optimization turned on (it's turned off by default in the stock build files), and the binaries could have been stripped of symbol names, although I'm not sure that helps with compiled C/C++ performance. Catalina could also be compiled without debugging information in the .class files, which does help a bit with performance (that's also turned off by default, since without line numbers it becomes tough to diagnose and fix any problems an administrator might run into. Also by running a class "obfuscator" on the compiled class files, most of the symbol names are reduced to three or four character names, which makes the classes smaller and faster. I will likely run more tests to see if turning all of this on for both will make either of them perform substantially better (I doubt that it will), and I encourage others to try the same. Conclusion The characteristics of each server are quite different. Apache 1.3 handles requests by preforking server processes in a pool of server processes, whereas Catalina uses Java threads all running in one process. Memory usage is different. But, the benchmark numbers show that the performance difference between the two servers (for my tests, anyway) is about 50%, meaning that each request took Catalina an average of about 90 milliseconds longer to serve. That's a difference of less than a tenth of a second. While Catalina's raw performance on static content isn't yet faster than Apache, It's not very far behind, and it's probably more than adequate for most uses. Many people using Tomcat are using it behind Apache (or another third- party web server). This means that they must compile a native code module into Apache (like mod_jserv, mod_jk, or the new mod_webapp) and run both Apache and Catalina simultaneously so that they can serve both static and dynamic content. While this setup is common and functional, it has the following drawbacks: 1) it takes up more memory -- they must run both servers instead of just one. 2) it creates a performance drain due to the inter-server communication that must happen per request. 3) if Catalina is set up on a machine other than the one on which Apache runs, all requests must again go over ethernet to get to Catalina. Even though this may be a fast internal net, it does slow down the request somewhat. This means they may lose most of the advantage of running Apache (for performance reasons) in the first place. 4) it is harder to maintain since you are running two different servers, plus a native code module that may have versioning conflicts and/or incompatibilities between the two servers. So, even though they may gain some features by running Apache as their first-contact web server, they may in fact lose some performance due to running both servers and sending requests over the link between the two. And, there are more drawbacks to this setup that I haven't listed above. More benchmarking would be helpful, especially with a different number of concurrent clients, and even with different kinds of content (both static and dynamic). Special thanks to Roy Wilson who pointed out ApacheBench to all of us on the tomcat-dev list, and also to those who have written all of this great open source software!
