yeah this came out spectacularly, try out the patch at:
http://www.sqlalchemy.org/trac/attachment/ticket/2516/2516.patch
great idea thanks for bringing it up. it should help performance as there's
less overhead when instance-level events aren't present.
On Jun 20, 2012, at 4:10 PM, Michael Bayer wrote:
> You'll have one _ListenerCollection per event per target object. So first
> here's a script to illustrate:
>
> from sqlalchemy.event import _ListenerCollection as old_collection
> from sqlalchemy import event
> from collections import defaultdict
>
> class _ListenerCollection(old_collection):
> canary = defaultdict(int)
> def __init__(self, parent, target_cls):
> _ListenerCollection.canary[(target_cls, parent.__name__)] += 1
> old_collection.__init__(self, parent, target_cls)
>
> event._ListenerCollection = _ListenerCollection
> from sqlalchemy import *
>
> metadata = MetaData()
>
> for i in xrange(60):
> Table('t%d' % i, metadata,
> Column('id', Integer, primary_key=True),
> *[
> Column("c%d" % j, Integer)
> for j in xrange(10)
> ]
> )
>
> for (cls, name), count in _ListenerCollection.canary.items():
> print "Class: %r Event name: %s Count: %s" % (cls.__name__, name, count)
> print "Total: %d" % sum(_ListenerCollection.canary.values())
>
> output here is:
>
> Class: 'PrimaryKeyConstraint' Event name: before_parent_attach Count: 60
> Class: 'Column' Event name: before_parent_attach Count: 660
> Class: 'Table' Event name: before_parent_attach Count: 60
> Class: 'PrimaryKeyConstraint' Event name: after_parent_attach Count: 60
> Class: 'Column' Event name: after_parent_attach Count: 660
> Class: 'Table' Event name: after_parent_attach Count: 60
> Total: 1560
>
> what's happening here is each time a schema object is attached to a parent,
> the before/after events fire off. The event mechanics use a system where a
> blank _ListenerCollection comes online as soon as that event's dispatch is
> accessed. The event system is architected primarily for speed, so that
> there aren't many conditionals right now to check "if _listenercollection
> isn't needed here, then don't create one", etc.
>
> Memory is not usually an issue for folks as long as it isn't growing
> unbounded, unless this is some kind of embedded use case which would be
> interesting, obviously our primary target is servers.
>
> Given that you have special memory needs, while I'm not sure why
> _ListenerCollection is the target here, as there's plenty of strings, lists
> and dictionaries stuck on each Column as well, there are likely ways to
> improve this behavior nonetheless.
>
> Also _ListenerCollection doesn't use __slots__.
>
> I've added http://www.sqlalchemy.org/trac/ticket/2516 which has a quick proof
> of concept and a description of a technique that might work more completely,
> since its true the _ListenerCollection shouldn't have to be created if no
> events have been set up yet. I may have something up in awhile if I don't
> get pulled onto something else.
>
>
>
>
>
>
>
> On Jun 20, 2012, at 3:12 PM, R. David Murray wrote:
>
>> We're running sqlalchemy 0.7b4 (for internal reasons that aren't
>> important here), but I've also tested this against 0.7.6(*). Both under
>> Python3.2.2.
>>
>> We have an application with about 60 tables in the (declarative) schema
>> and perhaps 600 columns. After startup of the application, there are
>> some 4000 event._ListenerCollection objects. While that class uses
>> __slots__, it also allocates an empty list and an empty set in its
>> __init__. I've managed to reduce our startup memory usage by 600K by
>> hacking _ListenerCollection to not pre-allocate these empty objects.
>>
>> Is it normal for this many _ListenerCollections objects (and presumably
>> a bunch of other Event related objects) to get pre-allocated, or are we
>> doing something wrong in our configuration of the schema?
>>
>> --RDM
>>
>> (*) 0.7.6 uses about 2MB more memory at ap startup than 0.7b4 does.
>> I haven't tried to investigate why; I just found this out when I ran my
>> tests against it before sending this post.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "sqlalchemy" group.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected].
>> For more options, visit this group at
>> http://groups.google.com/group/sqlalchemy?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/sqlalchemy?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
