Re: templating benchmarks...
On Wed, 13 Jun 2001, Perrin Harkins wrote:
wow. template toolkil took a big hit, there. (no mod_perl on
this list? hmm!)
This benchmark can be very non-representive. If you don't know how to
optimize each and every thing under test, you end up with unfair
benchmark and come to potentially wrong conclusions. Take TT, add compiled
template caching on the disk and shared TT object and I bet TT won't be at
the bottom.
I actually helped Joshua tune the TT example a little, and it using a cached
Template object and caching the templates used in the test in memory. The
slowness comes from the fact that it provides a major feature that the
others don't, and it is being exercised in this test. The magic dot
notation which allows templates to say foo.bar.baz, regardless of what kind
of data structure, object, or code ref foo, bar, and baz may be takes
a little more work. Whether it's a good idea or not is left as an exercise
to the reader, but I will say this: if Template Toolkit is the bottleneck in
your app's performance, you have either done some serious tuning or written
a really simple application (like this benchmark).
I stand corrected.
I've tried to provide a general note of not taking any numbers for
granted. Your explanation is as usual the great one :)
Nevertheless, it's good to see some numbers, if only to convince Andy to
finish his optimized XS version of the TT stash.
Yeah, it beats our apps' performance badly, with some 8000 calls to
_dotop() in TT in some rendered-data-heavy requests, since we use deeply
nested datastructures like foo.bar.baz. I'll probably have to rework this
if Andy doesn't come up with Stash written in XS. If you plan to work with
the current TT and are going to use lots of vars in loops, consider not to
use hashes deeper than one level, and if you still want to use them,
consider reducing the nesting level wherever possible in the templates by
doing [% baz = foo.bar.baz %] before diving into a loop where foo.bar.baz
will be constantly accessed. (of course relevant for loops with many
iterations).
Surpisingly though, TT is faster than pure Perl when it comes to writing
loops rendering data from relatively deeply nested data structures:
Benchmark: timing 2000 iterations of access_perl, access_tt...
access_perl: 7 wallclock secs ( 7.01 usr + 0.00 sys = 7.01 CPU) @
285.31/s (n=2000)
access_tt: 5 wallclock secs ( 4.19 usr + 0.05 sys = 4.24 CPU) @
471.70/s (n=2000)
Here is the benchmark I've used. (this code is a dream of calendar
programmers, since all months are 31-days long :-) :
use Time::HiRes qw( gettimeofday tv_interval );
use Template;
my $tt = Template-new();
my $data =
{
month_name = [1..12],
day_event = [ map {
[ map { {time=1,info=2} } 1..31 ]
} 1..12
],
};
use Benchmark;
timethese(2000,
{
access_perl = \access_perl,
access_tt = \access_tt,
});
sub access_tt{
my $output = '';
$tt-process(\*DATA, {data=$data},\$output);
}
sub access_perl{
my $output = '';
for my $m (0..11){
$output .= Month: $data-{month_name}[$m]\n;
for my $d (0..30){
my $event = $data-{day_event}[$m][$d];
$output .= Time: $event-{time}\n;
$output .= Info: $event-{info}\n;
}
}
}
__DATA__
[% FOR m = [0..11] %]
Month: [% data.month_name.$m %]
[% FOR d = [0..30] %]
[% event = data.day_event.$m.$d %]
Time: [% event.time %]
Info: [% event.info %]
[% END %]
[% END %]
Cool, huh?
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://apachetoday.com http://eXtropia.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: templating benchmarks...
wow. template toolkil took a big hit, there. (no mod_perl on this list? hmm!) This benchmark can be very non-representive. If you don't know how to optimize each and every thing under test, you end up with unfair benchmark and come to potentially wrong conclusions. Take TT, add compiled template caching on the disk and shared TT object and I bet TT won't be at the bottom. I actually helped Joshua tune the TT example a little, and it using a cached Template object and caching the templates used in the test in memory. The slowness comes from the fact that it provides a major feature that the others don't, and it is being exercised in this test. The magic dot notation which allows templates to say foo.bar.baz, regardless of what kind of data structure, object, or code ref foo, bar, and baz may be takes a little more work. Whether it's a good idea or not is left as an exercise to the reader, but I will say this: if Template Toolkit is the bottleneck in your app's performance, you have either done some serious tuning or written a really simple application (like this benchmark). Nevertheless, it's good to see some numbers, if only to convince Andy to finish his optimized XS version of the TT stash. - Perrin
Re: templating benchmarks...
Tom Lancaster [EMAIL PROTECTED] wrote: Absolutely. But I'd like to bring up something I've noticed in benchmarking 'real' sites: many, if not all, of the templating solutions appear to parse the whole of an html page. This is at least true of Apache::ASP and HTML::Mason, which I have used. Is it not ? Not really. They all cache the page in memory. It is not re-parsed every time. I have produced really dramatic differences in performance in a two-tier setup by judicious use of mod_include vs. wholesale proxying of pages with dynamic content through to the mod_perl/Apache::ASP server. [snip] Granted, I have other major bottlenecks involved: using Berkeley DB v1.x for session state, for one. Perhaps this explains some of it -- maybe the proxied header/footer requests never make session calls. I suspect that it's a combination of the database access and the network transfer. There is no difference in the amount of parsing going on, since it's all cached after the first time (per child). - Perrin
Re: templating benchmarks...
This benchmark can be very non-representive. If you don't know how to optimize each and every thing under test, you end up with unfair benchmark and come to potentially wrong conclusions. Take TT, add compiled template caching on the disk and shared TT object and I bet TT won't be at the bottom. In any case always remember that it's extremely hard to run a fair benchmark. I'd say it's almost impossible. The only fair benchmarking can be done if you know all the in's and out's of the 'things' under test and provide many benchmark tests each exploring a single property and not just 'one for all' benchmark. Of course it's a good thing to have benchmarks, but they all should be taken with a grain of salt. Absolutely. But I'd like to bring up something I've noticed in benchmarking 'real' sites: many, if not all, of the templating solutions appear to parse the whole of an html page. This is at least true of Apache::ASP and HTML::Mason, which I have used. Is it not ? I have produced really dramatic differences in performance in a two-tier setup by judicious use of mod_include vs. wholesale proxying of pages with dynamic content through to the mod_perl/Apache::ASP server. For example: In a situation with 1 lightweight frontend proxy and two backend mod_perl/Apache::ASP app servers ( with load distributed evenly using a patched mod_rewrite and its ability to select randomly from a list in a file ) , in one part of the site the dynamic headers and footers are generated by using !--#include virtual=/apps/include/pane.html?pane=headerlocation=$REQUEST_URI --, where the file being included is in fact proxied back to the app servers to receive content; in other parts of the site similarly simple pages are proxied in their entirety to the app servers. The results I can produce ( granted only with 'ab' ) are stunningly different: when I request the header and footer from the app servers using mod_include plus my modified mod_rewrite ( the stock version refuses to rewrite proxied requests ), I get up to 600 requests / second. When proxying the whole page through I get around 6 requests / second. Granted, I have other major bottlenecks involved: using Berkeley DB v1.x for session state, for one. Perhaps this explains some of it -- maybe the proxied header/footer requests never make session calls. I suspect that the wholesale parsing/eval-ing of html pages also plays a part. What do y'all think ? Tom _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://eXtropia.com/ http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: templating benchmarks...
On Fri, 8 Jun 2001, will trillich wrote:
On Thu, Jun 07, 2001 at 06:48:38AM +0200, Gerald Richter wrote:
regarding the tools that dovetail into the mod_perl paradigm,
who's got a comparison over relative performance (and other
strengths/weaknesses) of various templating methods?
There are various discussions on the mod_perl list about this topic in the
past (so take a look at the archives). Also there was an start to write such
a comparsion, but I am not aware that anybody has really finished it. The
only benchmarks I know are from Joshua. I append his mail below.
NOTE: While the hello.xxx benchmarks only prints Hello world, so they only
measures the startup overhead of the toolkit, the h2000.xxx tests tends a
little bit more towards a real application.
this is interesting information -- perhaps misleading to use the
microscopic hello world but still it gives a starting point:
here i sorted by hits-per-second
Test Name Test File Hits/sec Bytes/Hit
-- -- --
HTML statichello.html 1158.4 311 bytes
mod_include SSIhello.shtm 996.6 198 bytes
mod_caucho JSP hello.jsp 860.6 230 bytes
mod_perl handler hello.benc 852.6 196 bytes
mod_php PHPhello.php 734.8 225 bytes
Apache::Registry v2.01 CGI Raw hello_raw. 706.4 52 bytes
Apache::Dispatch v0.08 handler hello/worl 656.1 196 bytes
HTML::Template v2.0hello.htmp 567.2 198 bytes
Apache::SSI v2.16 hello.shtm 559.4 199 bytes
Template v2.00 Toolkit hello.tt522.1 198 bytes
Apache::Registry v2.01 CGI.pm hello.reg 458.5 216 bytes
HTML::Embperl v2.0a18 hello.epl 458.2 219 bytes
Apache::ASP v2.07 hello.asp 390.6 241 bytes
Apache::ePerl hello.eper 344.8 217 bytes
HTML::Mason v0.895 hello.mas 365.3 197 bytes
i bet CGI would be 200.0 or so... ?
and here's the handler actualy does some work set:
mod_caucho JSP 2000h2000.jsp 328.9 28964 byte
mod_php PHP 2000 h2000.php 261.8 28865 byte
HTML::Embperl v2.0a18 2000 h2000.epl 247.3 28809 byte
Apache::ASP v2.07 2000 h2000.asp 228.0 28997 byte
HTML::Mason v0.895 2000h2000.mas 222.9 28798 byte
Template v2.00 Toolkit 2000h2000.tt 55.6 2 byte
wow. template toolkil took a big hit, there. (no mod_perl on
this list? hmm!)
This benchmark can be very non-representive. If you don't know how to
optimize each and every thing under test, you end up with unfair
benchmark and come to potentially wrong conclusions. Take TT, add compiled
template caching on the disk and shared TT object and I bet TT won't be at
the bottom.
In any case always remember that it's extremely hard to run a fair
benchmark. I'd say it's almost impossible. The only fair benchmarking can
be done if you know all the in's and out's of the 'things' under test and
provide many benchmark tests each exploring a single property and not just
'one for all' benchmark.
Of course it's a good thing to have benchmarks, but they all should be
taken with a grain of salt.
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://apachetoday.com http://eXtropia.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
