Re: dPERINTERP/MY_CXT (was: speeding up XS_DBI_dispatch())

2012-02-17 Thread Dave Mitchell
On Fri, Feb 10, 2012 at 10:03:06AM -0800, Jan Dubois wrote: On Fri, 10 Feb 2012, Tim Bunce wrote: On Fri, Feb 10, 2012 at 03:27:24PM +, Dave Mitchell wrote: If anyone knows of a more elegant way to make a function from DBI.xs available to DBD:: code, please let me know! I hope

RE: dPERINTERP/MY_CXT (was: speeding up XS_DBI_dispatch())

2012-02-17 Thread Jan Dubois
On Fri, 17 Feb 2012, Dave Mitchell wrote: On Fri, Feb 10, 2012 at 10:03:06AM -0800, Jan Dubois wrote: On Fri, 10 Feb 2012, Tim Bunce wrote: On Fri, Feb 10, 2012 at 03:27:24PM +, Dave Mitchell wrote: If anyone knows of a more elegant way to make a function from DBI.xs available to DBD::

undefined symbol: mro_meta_init on 5.10.0 (was: speeding up XS_DBI_dispatch())

2012-02-13 Thread Tim Bunce
On Sun, Feb 05, 2012 at 09:04:23PM +, Dave Mitchell wrote: Anyway, attached is the final patch. I've tested it under 5.8.1, 5.8.9, 5.14.2 and 5.15.7, with various permutations of threaded builds. I've just noticed that DBI 1.617_901 has been failing on perl 5.10.0

Re: dPERINTERP/MY_CXT (was: speeding up XS_DBI_dispatch())

2012-02-13 Thread Tim Bunce
On Fri, Feb 10, 2012 at 03:27:24PM +, Dave Mitchell wrote: I'm assuming that these won't be applied until after the windows() fork issue is fixed, so think of this email more as a preview. The first fix is relatively straightforward, and is private to DBI.xs, since the PERINTERP macros

Re: speeding up XS_DBI_dispatch()

2012-02-11 Thread Martin J. Evans
On 11/02/2012 00:48, Jan Dubois wrote: On Fri, 10 Feb 2012, Dave Mitchell wrote: On Thu, Feb 09, 2012 at 11:10:17PM +0100, demerphq wrote: Well perls fork() relies on threaded perl so it could very easily be Dave's patch. Dave do you have access to a win32 build environment? I'm afraid not.

Re: speeding up XS_DBI_dispatch()

2012-02-11 Thread Tim Bunce
On Fri, Feb 10, 2012 at 04:48:38PM -0800, Jan Dubois wrote: On Fri, 10 Feb 2012, Dave Mitchell wrote: On Thu, Feb 09, 2012 at 11:10:17PM +0100, demerphq wrote: Well perls fork() relies on threaded perl so it could very easily be Dave's patch. Dave do you have access to a win32 build

Re: speeding up XS_DBI_dispatch()

2012-02-11 Thread demerphq
On 11 February 2012 14:55, Tim Bunce tim.bu...@pobox.com wrote: On Fri, Feb 10, 2012 at 04:48:38PM -0800, Jan Dubois wrote: MSVCRT!abort perl514!PerlEnvGetenv+0x13 [perlhost.h @ 462] DBI!dbi_bootinit+0x276 [DBI.xs @ 468] DBI!XS_DBI__clone_dbis+0x71 [DBI.c @ 4280] Anyways, I would suggest

Re: speeding up XS_DBI_dispatch()

2012-02-10 Thread Dave Mitchell
On Thu, Feb 09, 2012 at 11:10:17PM +0100, demerphq wrote: Well perls fork() relies on threaded perl so it could very easily be Dave's patch. Dave do you have access to a win32 build environment? I'm afraid not. -- All wight. I will give you one more chance. This time, I want to hear no

Re: speeding up XS_DBI_dispatch()

2012-02-10 Thread Martin J. Evans
On 10/02/12 10:01, Dave Mitchell wrote: On Thu, Feb 09, 2012 at 11:10:17PM +0100, demerphq wrote: Well perls fork() relies on threaded perl so it could very easily be Dave's patch. Dave do you have access to a win32 build environment? I'm afraid not. Have you any suggestion on how I could

Re: speeding up XS_DBI_dispatch()

2012-02-10 Thread Martin J. Evans
On 10/02/12 10:54, Tim Bunce wrote: On Thu, Feb 09, 2012 at 09:53:08PM +, Martin J. Evans wrote: Something between 1.616_901 and 1.616_902 changed which breaks fork() in DBI on Windows. Ah, 1.616_901 and 1.616_902! So unrelated to Dave's method cache work. See if this works: -

Re: dPERINTERP/MY_CXT (was: speeding up XS_DBI_dispatch())

2012-02-10 Thread Dave Mitchell
On Thu, Jan 26, 2012 at 12:26:44PM +, Tim Bunce wrote: On Wed, Jan 25, 2012 at 05:37:13PM -0800, Jan Dubois wrote: On Wed, 25 Jan 2012, Tim Bunce wrote: On Wed, Jan 25, 2012 at 04:14:07PM +, Dave Mitchell wrote: PS - I'm specifically being paid only to fix a performance issue

Re: dPERINTERP/MY_CXT (was: speeding up XS_DBI_dispatch())

2012-02-10 Thread Tim Bunce
On Fri, Feb 10, 2012 at 03:27:24PM +, Dave Mitchell wrote: On Thu, Jan 26, 2012 at 12:26:44PM +, Tim Bunce wrote: I attach two patches; the first replaces dPERINTERP with dMY_THX, while the second extends this to the DBIS stuff too. The headline figure (YMMV etc) is that on a

RE: dPERINTERP/MY_CXT (was: speeding up XS_DBI_dispatch())

2012-02-10 Thread Jan Dubois
On Fri, 10 Feb 2012, Tim Bunce wrote: On Fri, Feb 10, 2012 at 03:27:24PM +, Dave Mitchell wrote: If anyone knows of a more elegant way to make a function from DBI.xs available to DBD:: code, please let me know! I hope to take a proper look in the next day or so. The standard way

RE: speeding up XS_DBI_dispatch()

2012-02-10 Thread Jan Dubois
On Fri, 10 Feb 2012, Dave Mitchell wrote: On Thu, Feb 09, 2012 at 11:10:17PM +0100, demerphq wrote: Well perls fork() relies on threaded perl so it could very easily be Dave's patch. Dave do you have access to a win32 build environment? I'm afraid not. The problem is due to the getenv()

Re: speeding up XS_DBI_dispatch()

2012-02-09 Thread demerphq
On 9 February 2012 22:53, Martin J. Evans martin.ev...@easysoft.com wrote: Something between 1.616_901 and 1.616_902 changed which breaks fork() in DBI on Windows. The test 16destroy.t fails with a Windows error This application has requested the Runtime to terminate in an unusual way

Re: speeding up XS_DBI_dispatch()

2012-02-09 Thread Tim Bunce
On Thu, Feb 09, 2012 at 09:53:08PM +, Martin J. Evans wrote: Something between 1.616_901 and 1.616_902 changed which breaks fork() in DBI on Windows. The test 16destroy.t fails with a Windows error This application has requested the Runtime to terminate in an unusual way I'd guess

Re: speeding up XS_DBI_dispatch()

2012-02-09 Thread Martin J. Evans
On 09/02/2012 22:12, Tim Bunce wrote: On Thu, Feb 09, 2012 at 09:53:08PM +, Martin J. Evans wrote: Something between 1.616_901 and 1.616_902 changed which breaks fork() in DBI on Windows. The test 16destroy.t fails with a Windows error This application has requested the Runtime to

Re: speeding up XS_DBI_dispatch()

2012-02-08 Thread Dave Mitchell
On Wed, Feb 08, 2012 at 10:58:37AM +, Tim Bunce wrote: PS - I'm currently working on bringing PERINTERP and DBIS into the Brave New World of MY_CXT. Wonderful! I presume that'll cause an API change that'll require compiled drivers to be recompiled (ie. we'd bump DBISTATE_VERSION so

Re: speeding up XS_DBI_dispatch()

2012-02-07 Thread Tim Bunce
On Sun, Feb 05, 2012 at 09:04:23PM +, Dave Mitchell wrote: On Mon, Jan 30, 2012 at 10:56:37PM +, Tim Bunce wrote: dbih_getcom2 does the first part of that when looking up DBI hande magic. I never added the move to the top because I figured it would be very rare for magic to get

Re: speeding up XS_DBI_dispatch()

2012-02-07 Thread Dave Mitchell
On Tue, Feb 07, 2012 at 10:54:12PM +, Tim Bunce wrote: -if (!imp_msv) { +if (!imp_msv || !GvCV(imp_msv)) { What's the GvCV test for? One of the more vicious tests in t/31methcache.t does local $DBD::Sponge::st::{fetch}; which causes gv_fetchmeth(...,'fetch') to

Re: speeding up XS_DBI_dispatch()

2012-01-30 Thread Dave Mitchell
On Sun, Jan 29, 2012 at 10:06:58PM +, Tim Bunce wrote: On Sun, Jan 29, 2012 at 06:12:50PM +, Dave Mitchell wrote: The code itself (see diff below) just attaches some magic to the outer CV that caches the GV of the corresponding inner method. Any reason for not using the existing

Re: speeding up XS_DBI_dispatch()

2012-01-30 Thread Tim Bunce
On Mon, Jan 30, 2012 at 12:48:32PM +, Dave Mitchell wrote: On Sun, Jan 29, 2012 at 10:06:58PM +, Tim Bunce wrote: On Sun, Jan 29, 2012 at 06:12:50PM +, Dave Mitchell wrote: The code itself (see diff below) just attaches some magic to the outer CV that caches the GV of the

Re: speeding up XS_DBI_dispatch()

2012-01-29 Thread Dave Mitchell
On Wed, Jan 25, 2012 at 04:14:07PM +, Dave Mitchell wrote: then the CPU usage of the while loop broke down as follows: 7.0% overhead of loop, i.e. while() {$c++} 19.2% handle the outer method call, i.e. $sth-fetch() of which half is method lookup,

Re: speeding up XS_DBI_dispatch()

2012-01-29 Thread Tim Bunce
On Sun, Jan 29, 2012 at 06:12:50PM +, Dave Mitchell wrote: I've now written some working caching code, that reduces CPU usage on while ($sth-fetch()) {$c++} from 15.23s to 13.10s, which is a 14% saving!! The test code just fetches millions of rows from a 2-int table and then doesn't do

Re: speeding up XS_DBI_dispatch()

2012-01-27 Thread Tim Bunce
On Wed, Jan 25, 2012 at 08:17:27PM +, Tim Bunce wrote: On Wed, Jan 25, 2012 at 04:14:07PM +, Dave Mitchell wrote: The DBI is always allowed to cheat in the name of performance! My simple but effective xsbypass hack doesn't seem to have caused problems in the years it's been

Re: speeding up XS_DBI_dispatch()

2012-01-27 Thread Dave Mitchell
On Fri, Jan 27, 2012 at 02:53:47PM +, Tim Bunce wrote: On Wed, Jan 25, 2012 at 08:17:27PM +, Tim Bunce wrote: On Wed, Jan 25, 2012 at 04:14:07PM +, Dave Mitchell wrote: Just out of curiosity, why can't threaded perls use it? To be honest I can't remember now. It's disabled

Re: speeding up XS_DBI_dispatch()

2012-01-27 Thread Tim Bunce
On Fri, Jan 27, 2012 at 03:33:51PM +, Dave Mitchell wrote: On Fri, Jan 27, 2012 at 02:53:47PM +, Tim Bunce wrote: On Wed, Jan 25, 2012 at 08:17:27PM +, Tim Bunce wrote: On Wed, Jan 25, 2012 at 04:14:07PM +, Dave Mitchell wrote: Just out of curiosity, why can't threaded

dPERINTERP/MY_CXT (was: speeding up XS_DBI_dispatch())

2012-01-26 Thread Tim Bunce
On Wed, Jan 25, 2012 at 05:37:13PM -0800, Jan Dubois wrote: On Wed, 25 Jan 2012, Tim Bunce wrote: On Wed, Jan 25, 2012 at 04:14:07PM +, Dave Mitchell wrote: PS - I'm specifically being paid only to fix a performance issue on non-threaded builds, so I won't be looking at threaded

Re: speeding up XS_DBI_dispatch()

2012-01-25 Thread Dave Mitchell
On Tue, Jan 24, 2012 at 08:42:28PM +, Tim Bunce wrote: On Mon, Jan 23, 2012 at 12:51:12AM +, Dave Mitchell wrote: In the particular case I've been investigating, the method lookup in XS_DBI_dispatch() contributed a measurable part of the total loop time; in particular, when as a

Re: speeding up XS_DBI_dispatch()

2012-01-25 Thread Tim Bunce
On Wed, Jan 25, 2012 at 04:14:07PM +, Dave Mitchell wrote: On Tue, Jan 24, 2012 at 08:42:28PM +, Tim Bunce wrote: Given that my hacky cache for fetch() lookup reduced overall execution time by 16.5%, that reduces overall XS_DBI_dispatch percentage from 33.2% to 16.7%, i.e. nearly

RE: speeding up XS_DBI_dispatch()

2012-01-25 Thread Jan Dubois
On Wed, 25 Jan 2012, Tim Bunce wrote: On Wed, Jan 25, 2012 at 04:14:07PM +, Dave Mitchell wrote: PS - I'm specifically being paid only to fix a performance issue on non-threaded builds, so I won't be looking at threaded issues. But dPERINTERP looks like bad news on threaded builds.

speeding up XS_DBI_dispatch()

2012-01-22 Thread Dave Mitchell
[ disclaimer: I am being funded to do this work ] I've been looking into bottlenecks in a tight fetch loop with millions of rows, e.g. $sth-bind_columns(...); while ($sth-fetch) { # do a small amount of work } In the particular case I've been investigating, the method lookup