You'll have one _ListenerCollection per event per target object. So first
here's a script to illustrate:
from sqlalchemy.event import _ListenerCollection as old_collection
from sqlalchemy import event
from collections import defaultdict
class _ListenerCollection(old_collection):
canary = defaultdict(int)
def __init__(self, parent, target_cls):
_ListenerCollection.canary[(target_cls, parent.__name__)] += 1
old_collection.__init__(self, parent, target_cls)
event._ListenerCollection = _ListenerCollection
from sqlalchemy import *
metadata = MetaData()
for i in xrange(60):
Table('t%d' % i, metadata,
Column('id', Integer, primary_key=True),
*[
Column("c%d" % j, Integer)
for j in xrange(10)
]
)
for (cls, name), count in _ListenerCollection.canary.items():
print "Class: %r Event name: %s Count: %s" % (cls.__name__, name, count)
print "Total: %d" % sum(_ListenerCollection.canary.values())
output here is:
Class: 'PrimaryKeyConstraint' Event name: before_parent_attach Count: 60
Class: 'Column' Event name: before_parent_attach Count: 660
Class: 'Table' Event name: before_parent_attach Count: 60
Class: 'PrimaryKeyConstraint' Event name: after_parent_attach Count: 60
Class: 'Column' Event name: after_parent_attach Count: 660
Class: 'Table' Event name: after_parent_attach Count: 60
Total: 1560
what's happening here is each time a schema object is attached to a parent, the
before/after events fire off. The event mechanics use a system where a blank
_ListenerCollection comes online as soon as that event's dispatch is accessed.
The event system is architected primarily for speed, so that there aren't
many conditionals right now to check "if _listenercollection isn't needed here,
then don't create one", etc.
Memory is not usually an issue for folks as long as it isn't growing unbounded,
unless this is some kind of embedded use case which would be interesting,
obviously our primary target is servers.
Given that you have special memory needs, while I'm not sure why
_ListenerCollection is the target here, as there's plenty of strings, lists and
dictionaries stuck on each Column as well, there are likely ways to improve
this behavior nonetheless.
Also _ListenerCollection doesn't use __slots__.
I've added http://www.sqlalchemy.org/trac/ticket/2516 which has a quick proof
of concept and a description of a technique that might work more completely,
since its true the _ListenerCollection shouldn't have to be created if no
events have been set up yet. I may have something up in awhile if I don't get
pulled onto something else.
On Jun 20, 2012, at 3:12 PM, R. David Murray wrote:
> We're running sqlalchemy 0.7b4 (for internal reasons that aren't
> important here), but I've also tested this against 0.7.6(*). Both under
> Python3.2.2.
>
> We have an application with about 60 tables in the (declarative) schema
> and perhaps 600 columns. After startup of the application, there are
> some 4000 event._ListenerCollection objects. While that class uses
> __slots__, it also allocates an empty list and an empty set in its
> __init__. I've managed to reduce our startup memory usage by 600K by
> hacking _ListenerCollection to not pre-allocate these empty objects.
>
> Is it normal for this many _ListenerCollections objects (and presumably
> a bunch of other Event related objects) to get pre-allocated, or are we
> doing something wrong in our configuration of the schema?
>
> --RDM
>
> (*) 0.7.6 uses about 2MB more memory at ap startup than 0.7b4 does.
> I haven't tried to investigate why; I just found this out when I ran my
> tests against it before sending this post.
>
> --
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/sqlalchemy?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
