On Mon, Jun 11, 2007 at 07:39:08PM +1000, Malcolm Tredinnick wrote: > > On Sun, 2007-06-10 at 09:07 -0700, Brian Harring wrote: > > Curious, how many folks are actually using dispatch at all? > > > > For my personal usage, I'm actually not using any of the hooks- I > > suspect most folks aren't either. That said, I'm paying a fairly > > hefty price for them. > > > > With Model.__init__'s send left in for 53.3k record instantiation > > (just a walk of the records), time required is 9.2s. Without the > > send, takes 7.0s. Personally, I'd like to get that quarter of the > > time slice back. :) > > Since you already have your own version of the Spanish Inquisition set > up for testing, what portion of this overhead is just the function call? > If you the dispatch function is replaced with just "return", do we save > much.
Offhand, replacing the dispatch with just 'return' is actually
semi tricky, since there are a few receivers required for the django
internals (class preparation). Basically requires delegating the send
to the signal in select cases (for *_delete, and request_*, don't see
much option unless they can be shifted around also).
For __init__ and save however, the wrap trick will fly- meaning don't
even need the empty function call.
Either way, profile dump follows.
Top 30 via lsprof (cProfile for 2.5); with send left in Model.__init__
>>> ps.sort_stats("ti").print_stats(30)
Mon Jun 11 02:55:18 2007 dump.stats
1747388 function calls (1745991 primitive calls) in 18.627 CPU seconds
Ordered by: internal time
List reduced from 916 to 30 due to restriction <30>
ncalls tottime percall cumtime percall filename:lineno(function)
53332 3.314 0.000 9.046 0.000 base.py:97(__init__)
106720 2.592 0.000 3.162 0.000 dispatcher.py:271(getAllReceivers)
373318 1.995 0.000 3.056 0.000 base.py:38(utf8)
2 1.723 0.861 1.723 0.861 base.py:99(execute)
53332 1.631 0.000 4.687 0.000 base.py:37(utf8rowFactory)
537 1.540 0.003 6.227 0.012 ~:0(<method 'fetchmany' of
'pysqlite2.dbapi2.Cursor' objects>)
380348 1.329 0.000 1.329 0.000 ~:0(<setattr>)
196950 1.061 0.000 1.061 0.000 ~:0(<method 'encode' of 'unicode'
objects>)
53334 0.809 0.000 17.958 0.000 query.py:171(iterator)
106678 0.808 0.000 3.980 0.000 dispatcher.py:317(send)
213374 0.570 0.000 0.570 0.000 ~:0(<id>)
116228/115981 0.321 0.000 0.322 0.000 ~:0(<len>)
3 0.168 0.056 18.126 6.042 query.py:468(_get_data)
53334 0.162 0.000 0.162 0.000 ~:0(<iter>)
60 0.060 0.001 0.118 0.002 functional.py:26(__init__)
245/60 0.042 0.000 0.158 0.003 sre_parse.py:374(_parse)
6660 0.036 0.000 0.036 0.000 functional.py:36(__promise__)
3118 0.035 0.000 0.051 0.000 sre_parse.py:182(__next)
458/56 0.032 0.000 0.126 0.002 sre_compile.py:27(_compile)
1 0.027 0.027 0.032 0.032
sre_compile.py:296(_optimize_unicode)
206 0.022 0.000 0.067 0.000
sre_compile.py:202(_optimize_charset)
7 0.021 0.003 0.455 0.065 __init__.py:1(?)
6360 0.017 0.000 0.017 0.000 ~:0(<method 'append' of 'list'
objects>)
2573 0.017 0.000 0.059 0.000 sre_parse.py:201(get)
634/243 0.014 0.000 0.018 0.000 sre_parse.py:140(getwidth)
1 0.014 0.014 18.627 18.627 full-run.py:2(?)
19/12 0.012 0.001 0.187 0.016 ~:0(<__import__>)
1 0.008 0.008 0.011 0.011 socket.py:43(?)
1 0.007 0.007 0.109 0.109 urllib2.py:71(?)
182/57 0.007 0.000 0.160 0.003 sre_parse.py:301(_parse_sub)
<pstats.Stats instance at 0xb7ce268c>
without
>>> ps.sort_stats("ti").print_stats(30)
Mon Jun 11 03:02:10 2007 /home/bharring/dump2.stats
1320732 function calls (1319335 primitive calls) in 13.970 CPU seconds
Ordered by: internal time
List reduced from 916 to 30 due to restriction <30>
ncalls tottime percall cumtime percall filename:lineno(function)
53332 2.428 0.000 4.475 0.000 base.py:97(__init__)
373318 2.004 0.000 3.059 0.000 base.py:38(utf8)
2 1.715 0.858 1.716 0.858 base.py:99(execute)
380348 1.623 0.000 1.623 0.000 ~:0(<setattr>)
53332 1.617 0.000 4.676 0.000 base.py:37(utf8rowFactory)
537 1.538 0.003 6.215 0.012 ~:0(<method 'fetchmany' of
'pysqlite2.dbapi2.Cursor' objects>)
196950 1.055 0.000 1.056 0.000 ~:0(<method 'encode' of 'unicode'
objects>)
53334 0.744 0.000 13.305 0.000 query.py:171(iterator)
116228/115981 0.315 0.000 0.317 0.000 ~:0(<len>)
3 0.166 0.055 13.470 4.490 query.py:468(_get_data)
53334 0.158 0.000 0.158 0.000 ~:0(<iter>)
60 0.060 0.001 0.119 0.002 functional.py:26(__init__)
245/60 0.042 0.000 0.158 0.003 sre_parse.py:374(_parse)
6660 0.036 0.000 0.036 0.000 functional.py:36(__promise__)
3118 0.035 0.000 0.051 0.000 sre_parse.py:182(__next)
458/56 0.032 0.000 0.127 0.002 sre_compile.py:27(_compile)
1 0.027 0.027 0.032 0.032
sre_compile.py:296(_optimize_unicode)
206 0.022 0.000 0.067 0.000
sre_compile.py:202(_optimize_charset)
7 0.021 0.003 0.455 0.065 __init__.py:1(?)
2573 0.017 0.000 0.059 0.000 sre_parse.py:201(get)
6360 0.017 0.000 0.017 0.000 ~:0(<method 'append' of 'list'
objects>)
634/243 0.014 0.000 0.018 0.000 sre_parse.py:140(getwidth)
1 0.014 0.014 13.970 13.970 full-run.py:2(?)
19/12 0.011 0.001 0.188 0.016 ~:0(<__import__>)
1 0.008 0.008 0.011 0.011 socket.py:43(?)
182/57 0.007 0.000 0.160 0.003 sre_parse.py:301(_parse_sub)
1 0.007 0.007 0.109 0.109 urllib2.py:71(?)
1456 0.007 0.000 0.015 0.000 sre_parse.py:195(match)
206 0.007 0.000 0.076 0.000
sre_compile.py:173(_compile_charset)
20 0.006 0.000 0.007 0.000 sre_compile.py:253(_mk_bitmap)
Model.__init__ is still a bit of a kick in the teeth offhand;
addressing that one however requires some semi-nasty work shifting
some of the fields related testing to be cached in _meta; not
expecting a huge gain out of it, plus it'll likely be fairly nasty so
I'd rather hold off on that one till a later date.
Not yet advocating it (mainly since digging it out would be ugly), but
if you take a look at the bits above, having the option to disable
verification on read *would* have a nice kick in the pants for ORM
object instantiation when the admin has decided the data is guranteed
to be the correct types.
> In case it's not clear: I'm trying to get a feeling for how much of the
> cost is caused by the dispatching itself and how much by processing the
> dispatch inside the signal module. Is avoiding the call altogether
> necessary or making the handlers much faster? (More for future direction
> than anything else).
Cost is from the dispatching; take a look in dispatcher.send. Django
codebase has already deviated from dispatcher upstream via inlining
large parts of the lookup there (part of the 5x boost in dispatching
going from 0.95 to 0.96)- still has to do the lookups, which
unfortunately are semi complex due to the semantics of Any.
That said... there really isn't any reason to continue making the
calls if you know nothing is listening and the target to wrap emits
just pre/post.
> > Aside from that, would really help if I had a clue what folks are
> > actually using dispatch for with django- which signals, common
> > patterns for implementing their own signals (pre/post I'd assume?),
> > common signals they're listening to, etc.
>
> I don't think we can hope to get a really accurate picture here beyond a
> statistical sample with a broad range for any real confidence interval.
> The problem is that there is a userbase of thousands and a lot of
> evidence to suggest that most people don't read mailing list threads
> that they didn't start themselves. Yes, almost everybody reading this is
> an exception, but that automatically makes you an outlier.
>
> However, to add to the sample, I'm using post_init in some cases and
> pre_save and post_save a lot. Looks like request_started and
> request_finished are making an appearance in my code, but mostly in
> diagnostic stuff that is not intended for production use.
Just looking to get an idea of what folks are actually doing; simple
example, it's easier to fire both pre/post if there is a listener for
one- that said, if the vast number of folks are listening to only
*one* of the signals, it's potentially worth the time to have the code
swap in a pre, pre + post, or post wrapper as needed.
Also is a bit more of a pain in the ass implementing that, but looks
of it, it'll be the desired next step.
> > Knowing it would help with optimizing dispatch further, and would be
> > useful if someone ever decides to gut dispatch and refactor the code
> > into something less fugly.
>
> Given that upstream pydispatcher isn't really being maintained, I don't
> think we should be too hesitant to tweak it for our needs.
Don't spose it could just be thrown out? The code really *is* ugly :)
Can likely drop a lot of the internal voodoo and shift over to using
weakref.Weak*Dictionary where appropriate internally, but robustapply
is still fairly nasty- tend to think it should stop trying to hold
folks hands, and just pass the send args/kwargs straight through to
the receiver instead of trying to map args out.
~harring
pgp4XVwdljvUq.pgp
Description: PGP signature
