On Fri, Feb 16, 2007 at 03:13:35AM -0800, Ask Bjorn Hansen wrote:
>
> On Feb 14, 2007, at 3:10 PM, Tim Bunce wrote:
>
> >The http transport will enable not just true connection pooling via
> >apache's acting as 'middleware', but also load balancing and response
> >caching via hardware web accelerators. (Or Squid or whatever caching /
> >reverse proxy / load balancing technology you want to use.)
>
> I think Gearman can provide some of the same or similar benefits with
> potentially lower overhead.
Where would the lower overhead come from? Avoiding using of LWP modules?
Also, does gearman support caching of responses?
Anyway, speaking of overheads... I've rewritten t/85gofer.t so it tests
multiple transports with multiple policy modules. It also reports some
handy stats. Here's a summary of 2000 inserts and selects on my laptop
comparing the null and http transports:
insert ~baseline~ : 0.000425sec ( 2351/sec)
insert null+rush : 0.001554sec ( 643/sec) +1.1ms
insert null+pedantic: 0.001579sec ( 633/sec) +1.2ms
insert null+classic : 0.001568sec ( 637/sec) +1.1ms
insert http+rush : 0.004781sec ( 209/sec) +4.4ms
insert http+pedantic: 0.005440sec ( 183/sec) +5.0ms
insert http+classic : 0.004704sec ( 212/sec) +4.3ms
select ~baseline~ : 0.000550sec ( 1818/sec)
select null+rush : 0.001702sec ( 587/sec) +1.2ms
select null+pedantic: 0.002805sec ( 356/sec) +2.3ms
select null+classic : 0.001722sec ( 580/sec) +1.2ms <===
select http+rush : 0.004510sec ( 221/sec) +4.0ms
select http+pedantic: 0.008916sec ( 112/sec) +8.4ms
select http+classic : 0.004507sec ( 221/sec) +4.0ms <===
So the core DBD::Gofer overhead (the null transport) is ~1.2ms per request,
and the http transport adds a further ~2.8ms.
I've checked it in. I'd be very interested to see the results for gearman.
> >>To try it,
> >[...]
> >
> >Better than that... just set the DBI_AUTOPROXY env var to
> >
> > dbi:Gofer:transport=gearman;url=127.0.0.1
> >
> >and do "make test", then the *whole DBI test suite* will be run
> >through it!
>
> When I only run one worker / transport end-point (what's the lingo
> for this?)
Good question. I'm not very comfortable with the current (lack of)
terminology but I've not given much though to picking new words yet.
> it seems to get stuck on executing a (random?) "ping" request.
>
> If I run two workers/end-points then all tests but the 85gofer.t test
> succeed with DBI_AUTOPROXY.
Sounds like gearman issues. I've not seen that issue with the http transport.
> To my surprise it's almost as fast to run the test suite "directly"
> and "through gearman" - 24 vs 27 wall clock seconds on my system.
That does seem a little odd. On my laptop it takes ~42 seconds normally
and ~56 when run with DBI_AUTOPROXY=dbi:Gofer:transport=null. So about
25% slower even with the null transport. So to be only ~12% slower via
gearman would be surprising. Worth rechecking.
> >>Index: lib/DBD/Gofer/Transport/gearman.pm
> >
> >Ask, go ahead and check it in to the repository. With a few changes...
>
> Ok - I'll do that over the weekend when I've had a chance to clean it
> up a bit.
Great. I'm aiming to get a release candidate out today for feedback on
the http transport and other changes. There are more docs and tweaks to
be done so I'd expect another release candidate early next week.
Would be great to have gearman incuded in that.
> >>+package DBI::Gofer::Transport::gearman;
> >[...]
> >
> >You could optimize that slighly by reusing the same $executor object.
>
> Yup, that saved two wall clock seconds ("make test" with autoproxy
> went from ~29-30 to 27 seconds).
Small improvements add up fast when they're executed thousands of times.
> I started out writing it to support different executor configs (like
> the mod_perl target), but I realized that with gearman you probably
> would just run different workers anyway and have them register
> different "methods" for the different database targets.
FYI The mod_perl (server-side) transport httpd.conf currently looks like:
<Location /gofer>
SetHandler perl-script
PerlHandler DBI::Gofer::Transport::mod_perl
</Location>
or, more generally:
<Location /gofer/example>
SetHandler perl-script
PerlSetVar GoferConfig default
PerlAddVar GoferConfig example
PerlHandler DBI::Gofer::Transport::mod_perl
</Location>
Where the optional GoferConfig 'default' and 'example' clauses refer to
named configurations you can setup like this:
PerlModule DBI::Gofer::Transport::mod_perl
<Perl>
DBI::Gofer::Transport::mod_perl->configuration({
default => {
default_connect_dsn => '...',
default_connect_attributes => { # use these if corresponding
attribute not provided
Username => 'anonymous',
Password => 'foobar',
},
forced_connect_attributes => { # force use of these values
},
},
example => {
forced_connect_dsn => 'dbi:Sybase:example',
default_connect_attributes => { # use these if corresponding
value not provided
ChopBlanks => 1,
},
forced_connect_attributes => { # force use of these values
}
}
});
</Perl>
Where a Location refers to multiple GoferConfig names they are merged.
Individual settings in later configs override the same in earlier ones.
So, for example, requests to /gofer/example would always connect to
'dbi:Sybase:example' because of the forced_connect_dsn setting.
And the username and password would default to anonymous/foobar
if none were supplied in the request.
Suggestions most welcome on this, or anything else for that matter...
Tim.
