On Jan 13, 2013, at 12:40 AM, Anoop K wrote: > Hi, > > I had POSTed a > question(https://groups.google.com/forum/?fromgroups=#!topic/sqlalchemy/vCjrGE9zYwQ) > related to session/engine cleanup. > > After fixing the engine cleanup. I could see that engine and its associated > objects are GCed properly. However > process memory is gradually increasing. I am creating and destroying > session/engine every 30 secs. > > SQLAlchemy usage: > > Every 30 sec following happens. > > engine = create_engine(...) > sn = Session(bind=engine)... > try: > sn.merge > sn.commit > finally: > sn.close() > sn.bind.dispose() > >>> sn and engine don't have any reference after this point .... > > > GC objects after some runs are listed below, Looks like some items are > increasing in memory(in BOLD). > Is this some kind of leak ?
First off, there's no way to know without a real, reproducing test case. Providing this would give us something tangible to examine. As the pattern of creating and destroying engines many times is not the primary use case for the Engine, it's hard to say definitively, though we do have memory tests which test the pattern of engines being created and disposed many times, at the same time testing the length of gc.get_objects() to confirm no growth in memory. In your example below, the objects in question are mostly type objects. These objects do have a behavior in that as created, they will cache information about themselves within a dialect that is local to an Engine. But as each Engine is disposed, so is the Dialect, and so is the cache. The cache itself is also a WeakKeyDictionary, so that no strong reference to the type object is created. So the cache does not prevent the type objects from being garbage collected. There are tests which ensure this behavior as well. In your counts below, there are 31 Dialect instances in one run and then 38 in another. So this suggests that there are 31 and then 38 Engine instances, a growth of 7, and that the Engine references are not in fact being cleaned out, or gc.collect() is not being called within the test. Depending on what you're doing with SQL expressions within each engine it is plausible that a few hundred type objects have accumulated in the cache. Note that it is absolutely necessary to call gc.collect() in any Python program that is attempting to get an accurate count of how many objects are still strongly referenced, as unreachable cycles can remain for a significant period of time if one waits for cyclic GC to happen automatically. > > <class 'sqlalchemy.util._collections.OrderedDict'>: last:176 curr:198 diff:22 > <type 'builtin_function_or_method'>: last:900 curr:914 diff:14 > <class 'sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2'>: last:31 > curr:38 diff:7 > <class 'sqlalchemy.types.NullType'>: last:32 curr:39 diff:7 > <type 'instancemethod'>: last:605 curr:627 diff:22 > <class 'sqlalchemy.types.Text'>: last:177 curr:254 diff:77 > <class 'sqlalchemy.dialects.postgresql.psycopg2._PGArray'>: last:120 > curr:197 diff:77 > <class 'sqlalchemy.sql.expression._BindParamClause'>: last:658 curr:954 > diff:296 > <class 'sqlalchemy.util._collections.PopulateDict'>: last:60 curr:82 diff:22 > <type 'list'>: last:3308 curr:3462 diff:154 > <type 'instance'>: last:139 curr:146 diff:7 > <class 'sqlalchemy.dialects.postgresql.base.PGTypeCompiler'>: last:31 > curr:38 diff:7 > <class > 'sqlalchemy.dialects.postgresql.psycopg2.PGIdentifierPreparer_psycopg2'>: > last:31 curr:38 diff:7 > <type 'dict'>: last:10930 curr:12172 diff:1242 > <class 'sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2'>: > last:60 curr:82 diff:22 > <class 'sqlalchemy.types.String'>: last:571 curr:753 diff:182 > <class 'sqlalchemy.types.DateTime'>: last:92 curr:113 diff:21 > <type 'weakref'>: last:2204 curr:2582 diff:378 > <type 'tuple'>: last:29850 curr:30231 diff:381 > <type 'cell'>: last:747 curr:978 diff:231 > <class 'sqlalchemy.types.Integer'>: last:52 curr:59 diff:7 > <type 'function'>: last:8077 curr:8392 diff:315 > > Thanks > Anoop > > > > -- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/sqlalchemy/-/gESaFAd9DHkJ. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
