Breakthrough: Using Gregg's expunge_all, then commenting out the addition to collections, SQLA is not leaking anymore.
Inside my "team" object, I go in then create the people and their statistics, but I don't add the people to the team collection anymore. The links are never used again, but it's a many-to-many relationship (a person can be on several teams), so I thought I needed to do this in order to get the many-to-many table working. Not putting these people in the team collections allows SQLA to free the resources. My question then becomes: how do I free up collections and have SQLA requery for those objects lazily? Chris On Apr 8, 4:18 pm, Chris Lewis <[email protected]> wrote: > Hi Gregg, > Changing the Session.remove() to an expunge_all doesn't have any > effect, the growth continues. > > Chris > > On Apr 8, 4:10 pm, Gregg Lind <[email protected]> wrote: > > > > > I believe you want session.expunge() or session.expunge_all(). > > > GL > > > On Wed, Apr 8, 2009 at 6:05 PM, Chris Lewis <[email protected]> wrote: > > > > Hi everyone! > > > I have a threaded application which deals with a lot of records (it > > > generates hundreds of thousands in an hour). I have a Database module > > > that my program imports, and use scoped_session to make the session > > > thread-safe. When a record is to be inserted, my program passes the > > > object to be saved to the Database insert() function. Here's the code: > > > > ============== > > > > engine = sa.create_engine(engine_url, echo=echo) > > > Session = sa.orm.scoped_session(sa.orm.sessionmaker(bind=engine, > > > autocommit=False)) > > > > def insert(obj): > > > try: > > > session().merge(obj) > > > except Exception, e: > > > log.warning("Database problem: " + str(e)) > > > session().rollback() > > > raise > > > else: > > > log.debug("Saved to database") > > > > session().commit() > > > Session.remove() > > > > def session(): > > > return Session() > > > > ============== > > > > Even though I call Session.remove(), it seems that I can't stop > > > sqlalchemy.orm.identity.IdentityManagedState growing. Unit testing > > > which inserts a couple thousand records shows the growth with Heapy. > > > The "dict of sqlalchemy.orm.identity.IdentityManagedState" starts at > > > 334 objects, ending with 11210 objects. > > > > I thought Session.remove() would cause SQLAlchemy to release those > > > resources, but this doesn't seem to be the case. As the process is > > > going to need to long-running (weeks hopefully), I'm far happier with > > > performing expensive CPU operations than exhausting my memory. > > > > I am certain this is my own error, but I am not sure what it is. Any > > > help would be appreciated! > > > > Thanks in advance, > > > Chris Lewis --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~----------~----~----~----~------~----~------~--~---
