Hey, [[ NUMBERS ARE BELOW ]] It has been a while, but here's a new set of Hello World benchmarks! What took me so long in getting these out is that the java web environments that I had set up would keep crashing during the tests in ways that would not only render their benchmarks meaningless, but also skewed the results for other tests, say when a java thread would spin out of control!
You can download the latest benchmark suite to run on your system at: http://www.chamas.com/bench/hello.tar.gz It seems to run on linux & solaris just fine, though the tests that are run depend on what environments you have set up. Please see the distribution README on how to run the suite for yourself. --Josh METHODOLOGY: These latest numbers come from my running the pubbench.sh script: ]# cat pubbench.sh #!/bin/bash ./bench.pl -version --ram --init-exec="perl javainit.pl -restart" -time=180 --store=results perl ./javainit.pl -stop The javainit.pl will kill any of my existing java environments, and if the current test needs java will start them up. The test run for 180 seconds total in 60 second batches, and for each 60 second run the javainit.pl script will be executed. The --ram flag attempts to calculate the RAM used by each test on my linux system during each 60 second test with the results averaged across each, so for applications with memory leaks, the RAM used only shows 60 seconds worth of them. The RAM calculation looks at RAM used & SWAP used from the linux /proc/meminfo file... as it looks at RAM before the httpd is fired up, but after the init-exec script executes, the RAM does not take into account the java web server environments being started, just memory usage that occurs after they start being used. DISCLAIMER: These numbers are just that, numbers. Please don't let them upset you. If you have a positive contribution that you would like to make, you may download the source code and submit patches ( preferably as diff -u ) to the test suite. This benchmark suite has evolved over years of benchmarking, and takes the point of view that web environments can only be well compared when run on the exact same system configuration... this is why the suite is done is such a way that you can run it on _your_ system giving you results relevant to you! NUMBERS: And without further adieu ... ( see NOTES below ) Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB) --------- --------- --------- --------- --------- --------- --------- --------- Apache::ASP v2.29 2000 h2000.asp 216.8 39042 180.09 0.004613 28998 33632 Apache::Registry v2.01 2000 mod_per h2000.reg 341.3 61442 180.04 0.002930 28179 16081 HTML::Embperl v1.3.0 2000 h2000.epl 113.2 20391 180.12 0.008833 28841 20418 HTML::Mason v1.03 2000 h2000.mas 84.0 15115 180.03 0.011911 28799 112220 HTML::Template v2.4 2000 h2000.htm 99.1 17848 180.03 0.010087 29152 50292 mod_caucho JSP 2000 h2000.jsp 97.6 17617 180.54 0.010248 28965 14057 mod_jserv JSP 2000 h2000.jsp 148.7 26773 180.04 0.006725 29408 29613 mod_php PHP 2000 h2000.php 258.2 46477 180.02 0.003873 28866 10212 Template v2.04 Toolkit 2000 h2000.tt 53.3 9604 180.03 0.018745 28889 55629 Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB) --------- --------- --------- --------- --------- --------- --------- --------- Apache::ASP v2.29 hello.asp 349.8 62985 180.05 0.002859 242 29601 Apache::Dispatch v0.09 handler hello/wor 585.8 105443 180.01 0.001707 197 9205 Apache::ePerl hello.epe 347.1 62508 180.08 0.002881 218 17477 Apache::Registry v2.01 CGI Raw hello.cgi 677.7 122003 180.03 0.001476 52 13061 Apache::Registry v2.01 CGI.pm hello.cgi 449.7 80967 180.05 0.002224 217 20717 Apache::SSI v2.16 hello.sht 546.7 98426 180.03 0.001829 200 13061 CGI::SpeedyCGI CGI Raw hello.scg 159.6 28722 180.00 0.006267 197 9748 CGI::SpeedyCGI CGI.pm hello.scg 145.1 26117 180.01 0.006892 217 10648 HTML static hello.htm 879.2 150000 170.61 0.001137 312 1982 HTML::Embperl v1.3.0 hello.epl 463.6 83471 180.05 0.002157 221 17333 HTML::Mason v1.03 hello.mas 368.5 66348 180.04 0.002714 198 29009 HTML::Template v2.4 hello.htm 539.1 97056 180.03 0.001855 199 14850 mod_caucho JSP hello.jsp 147.6 26596 180.20 0.006775 231 14442 mod_cgi CGI Raw hello.cgi 156.7 28208 180.02 0.006382 197 9506 mod_cgi CGI.pm hello.cgi 11.1 2040 183.53 0.089966 217 25893 mod_include SSI hello.sht 241.8 43605 180.36 0.004136 199 4297 mod_jserv JSP hello.jsp 73.1 13209 180.81 0.013688 358 10788 mod_perl handler hello.ben 813.8 146490 180.00 0.001229 197 5420 mod_php PHP hello.php 687.5 123753 180.00 0.001455 226 6008 Template v2.04 Toolkit hello.tt 469.2 84470 180.02 0.002131 199 42472 Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB) --------- --------- --------- --------- --------- --------- --------- --------- Apache::ASP v2.29 XSLT Hello hxslt.xml 185.0 33292 180.00 0.005407 292 39892 AxKit v1.4 XSLT Hello hxslt.xml 298.2 53678 180.03 0.003354 411 30622 Cocoon 1.8 mod_jserv XSLT Hello hxslt.xml 56.1 10144 180.80 0.017823 507 15784 Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB) --------- --------- --------- --------- --------- --------- --------- --------- Apache::ASP v2.29 XSLT Big hxsltbig. 138.6 24957 180.03 0.007214 21521 50572 AxKit v1.4 XSLT Big hxsltbig. 262.1 47206 180.08 0.003815 21859 31390 Cocoon 1.8 mod_jserv XSLT Big hxsltbig. 79.2 9525 120.25 0.012625 21921 29116 Apache Server Header Tokens --------------------------- (Unix) Apache/1.3.20 AxKit/1.4 OpenSSL/0.9.6a PHP/4.0.3pl1 Resin/1.2.1 mod_perl/1.25 mod_ssl/2.8.4 tomcat/1.0 PERL Version: 5.00503 JAVA Version: java version "1.3.0" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0) Classic VM (build 1.3.0, J2RE 1.3.0 IBM build cx130-20001124 (JIT enabled: jitc)) Operating System: Linux 2.2.14-9.0 (root@gate) (gcc egcs-2.91.66) #1 2CPU [gate.] NOTES: h2000.epl - Embperl 2.0 is much faster on this test, about the same speed as PHP, but its not installable from CPAN which is why I don't run it. When the release stablizes ( from alpha/beta release ), I will run 2.0 for this test. h2000.mas - HTML::Mason seems to have a memory leak on this test where it does not seem to have it on the hello.mas. If someone has a patch for the benchmark to fix this memory leak which would be consistent with ordinary Mason programming methods, please submit it. If there is a real memory leak under Mason for this test, then perhaps this test will help the authors fix it? h2000.jsp - mod_jserv beats mod_caucho on this test, but results flip flop for hello.jsp. I am not sure why this is. The java environments tend to benchmark unpredicatably, even for longer tests. This may be a standard result though and would not surprise me. HTML Static - Static HTML should generally be much faster than any other test, but is only slightly faster than mod_perl on my system. This is likely because my box is a dual processor 450 PIII running Linux 2.2.14 ( Redhat 6.2 ), and this version of linux does not deal efficiently with many processors. I imagine a benchmark on a 2.4.x linux kernel on this platform would show more of a difference. Cocoon 1.8 mod_jserv XSLT Big - This test only ran for 120 seconds, instead of 180 that all the other tests ran for. This is because of the 3 60 seconds test, the java process core dumped on one of them, so we only have 120 seconds worth of data. Also, why this test performs better than the Cocoon hxslt.xml, I have no idea. AxKit & Apache::ASP XSLT tests - These tests used XML::Sablotron, though they could have used XML::LibXSLT. For these tests it does not matter, because the transformation ends up being cached to disk anyway. Apache::ASP v2.29 XSLT Big - This test uses a lot of memory. This is likely due to Apache::ASP treating XSLT XML data & XSL templates as ASP scripts, which involves first compiling them to perl bytecode, prior to execution and final XSLT rendering of their output. This will likely be fixed in later releases by Apache::ASP detecting at runtime when files are static & don't have <% %> ASP code blocks, allow it to bypass the perl bytecode compilation process, which will help particularly for static XSLT scenarios.
