On Mon, Jan 23, 2012 at 12:51:12AM +0000, Dave Mitchell wrote:
> [ disclaimer: I am being funded to do this work ]
>
> I've been looking into bottlenecks in a tight fetch loop with millions of
> rows, e.g.
>
> $sth->bind_columns(...);
> while ($sth->fetch) {
> # do a small amount of work
> }
>
> In the particular case I've been investigating, the method lookup in
> XS_DBI_dispatch() contributed a measurable part of the total loop time;
> in particular, when as a test I added very crude and hacky caching for the
> lookup of the inner fetch method, the execution time for the loop dropped
> from 23.0s to 19.2s.
I wonder what that is as a % change in the cost of dispatch.
> So, I would like to attempt to add a method cache to XS_DBI_dispatch()
> (and possibly see if anything else can be tweaked).
>
> My questions are:
>
> * Would such a patch be welcome in principle?
Of course! My middle name is speed. ;-)
Though obviously the cost of caching shouldn't significantly impact more
varied usage where the hit ratio is lower.
> * Any gotchas that I might not be aware of (I'm not familiar with DBI/DBD
> internals apart from what I've played with for the last few days).
Hard to say without seeing the patch.
The guts of the DBI are ancient and baroque with many dusty corners
were dragons may, or may not, be lurking.
> * What's the earliest perl we have to maintain compatibility with?
5.8.1, though 5.8.3 would be ok.
> * Does the test suite already have good coverage for method lookup
> (especially where methods are redefined or stashes modified, i.e. the
> sort of thing that would invalidate caches), or should I add some?
Please do.
> * can I rely on _install_method() being the only way that XS CVs can get
> to point to XS_DBI_dispatch()?
Um, interesting. The DBI goes back a long way and I've no idea what
games some drivers may play. On the other hand I'm happy to say "assume
yes and we'll see what, if anything, breaks".
A slight snag is that we'd have to rely on the driver test suites and
they almost certainly wont have good coverage for methods being
redefined etc. So really we'd be relying on end user code to find
problems, and (I'm guessing) the symptoms may be subtle. On the other
hand, redefining methods within drivers would be an odd thing to do.
So, still, assume yes.
> * To what extent (if any) is DBI allowed to cheat when it comes to
> bypassing the perl API (in particular I'm thinking of the method
> invalidation stuff which I don't think has always been part of the API).
The DBI is always allowed to cheat in the name of performance!
My simple but effective xsbypass hack doesn't seem to have caused
problems in the years it's been there (though it's a pity threaded perls
can't use it).
Thanks Dave!
Tim.
