I tried memory tests by *reusing engine *and memory is steady. I feel some memory is held up even after the release of engine.
On Sunday, 13 January 2013 12:21:10 UTC+5:30, Anoop K wrote: > > gc.collect() was not called in the last sample. Here is one where > gc.collect() is also called > > > <class 'sqlalchemy.util._collections.OrderedDict'>: last:131 curr:134 > diff:3 > <type 'builtin_function_or_method'>: last:862 curr:868 diff:6 > <class 'sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2'>: > last:18 curr:21 diff:3 > <class 'sqlalchemy.types.NullType'>: last:19 curr:22 diff:3 > <type 'instancemethod'>: last:561 curr:564 diff:3 > <class 'sqlalchemy.types.Text'>: last:234 curr:264 diff:30 > <class 'sqlalchemy.dialects.postgresql.psycopg2._PGArray'>: last:180 > curr:210 diff:30 > <class 'sqlalchemy.sql.expression._BindParamClause'>: last:585 curr:678 > diff:93 > <class 'sqlalchemy.util._collections.PopulateDict'>: last:18 curr:21 > diff:3 > <type 'list'>: last:2962 curr:2983 diff:21 > <type 'instance'>: last:197 curr:200 diff:3 > <class 'sqlalchemy.dialects.postgresql.base.PGTypeCompiler'>: last:18 > curr:21 diff:3 > <class > 'sqlalchemy.dialects.postgresql.psycopg2.PGIdentifierPreparer_psycopg2'>: > last:18 curr:21 diff:3 > <type 'dict'>: last:10470 curr:10854 diff:384 > <class 'sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2'>: > last:18 curr:21 diff:3 > <class 'sqlalchemy.types.String'>: last:493 curr:550 diff:57 > <class 'sqlalchemy.types.DateTime'>: last:53 curr:56 diff:3 > <type 'weakref'>: last:2165 curr:2297 diff:132 > <type 'tuple'>: last:29338 curr:29473 diff:135 > <type 'cell'>: last:933 curr:1023 diff:90 > <class 'sqlalchemy.types.Integer'>: last:57 curr:63 diff:6 > <type 'function'>: last:8139 curr:8262 diff:123 > > > > > On Sunday, 13 January 2013 11:45:11 UTC+5:30, Michael Bayer wrote: >> >> >> On Jan 13, 2013, at 12:40 AM, Anoop K wrote: >> >> Hi, >> >> I had POSTed a question( >> https://groups.google.com/forum/?fromgroups=#!topic/sqlalchemy/vCjrGE9zYwQ >> ) >> related to session/engine cleanup. >> >> After fixing the engine cleanup. I could see that engine and its >> associated objects are GCed properly. However >> process memory is gradually increasing. I am creating and destroying >> session/engine every 30 secs. >> >> SQLAlchemy usage: >> >> Every 30 sec following happens. >> >> *engine = create_engine(...)* >> *sn = Session(bind=engine)...* >> *try:* >> * sn.merge* >> * sn.commit* >> *finally:* >> * sn.close()* >> * sn.bind.dispose()* >> *>>> sn and engine don't have any reference after this point ....* >> * >> * >> * >> * >> GC objects after some runs are listed below, Looks like some items are >> increasing in memory(in BOLD). >> *Is this some kind of leak ?* >> >> >> First off, there's no way to know without a real, reproducing test case. >> Providing this would give us something tangible to examine. >> >> As the pattern of creating and destroying engines many times is not the >> primary use case for the Engine, it's hard to say definitively, though we >> do have memory tests which test the pattern of engines being created and >> disposed many times, at the same time testing the length of >> gc.get_objects() to confirm no growth in memory. In your example below, >> the objects in question are mostly type objects. These objects do have a >> behavior in that as created, they will cache information about themselves >> within a dialect that is local to an Engine. But as each Engine is >> disposed, so is the Dialect, and so is the cache. The cache itself is >> also a WeakKeyDictionary, so that no strong reference to the type object is >> created. So the cache does not prevent the type objects from being garbage >> collected. There are tests which ensure this behavior as well. >> >> In your counts below, there are 31 Dialect instances in one run and then >> 38 in another. So this suggests that there are 31 and then 38 Engine >> instances, a growth of 7, and that the Engine references are not in fact >> being cleaned out, or gc.collect() is not being called within the test. >> Depending on what you're doing with SQL expressions within each engine it >> is plausible that a few hundred type objects have accumulated in the cache. >> >> Note that it is absolutely necessary to call gc.collect() in any Python >> program that is attempting to get an accurate count of how many objects are >> still strongly referenced, as unreachable cycles can remain for a >> significant period of time if one waits for cyclic GC to happen >> automatically. >> >> >> >> >> <class 'sqlalchemy.util._collections.OrderedDict'>: last:176 curr:198 >> diff:22 >> <type 'builtin_function_or_method'>: last:900 curr:914 diff:14 >> <class 'sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2'>: >> last:31 curr:38 diff:7 >> <class 'sqlalchemy.types.NullType'>: last:32 curr:39 diff:7 >> <type 'instancemethod'>: last:605 curr:627 diff:22 >> <class 'sqlalchemy.types.Text'>: last:177 curr:254 diff:77 >> * <class 'sqlalchemy.dialects.postgresql.psycopg2._PGArray'>: last:120 >> curr:197 diff:77* >> *<class 'sqlalchemy.sql.expression._BindParamClause'>: last:658 curr:954 >> diff:296* >> <class 'sqlalchemy.util._collections.PopulateDict'>: last:60 curr:82 >> diff:22 >> <type 'list'>: last:3308 curr:3462 diff:154 >> <type 'instance'>: last:139 curr:146 diff:7 >> <class 'sqlalchemy.dialects.postgresql.base.PGTypeCompiler'>: last:31 >> curr:38 diff:7 >> <class >> 'sqlalchemy.dialects.postgresql.psycopg2.PGIdentifierPreparer_psycopg2'>: >> last:31 curr:38 diff:7 >> <type 'dict'>: last:10930 curr:12172 diff:1242 >> <class 'sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2'>: >> last:60 curr:82 diff:22 >> *<class 'sqlalchemy.types.String'>: last:571 curr:753 diff:182* >> * <class 'sqlalchemy.types.DateTime'>: last:92 curr:113 diff:21* >> <type 'weakref'>: last:2204 curr:2582 diff:378 >> <type 'tuple'>: last:29850 curr:30231 diff:381 >> <type 'cell'>: last:747 curr:978 diff:231 >> <class 'sqlalchemy.types.Integer'>: last:52 curr:59 diff:7 >> <type 'function'>: last:8077 curr:8392 diff:315 >> >> Thanks >> Anoop >> * >> * >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "sqlalchemy" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/sqlalchemy/-/gESaFAd9DHkJ. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/sqlalchemy?hl=en. >> >> >> -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To view this discussion on the web visit https://groups.google.com/d/msg/sqlalchemy/-/FPTcq6Jdf4AJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
