Hi,
I'm using an older copy of SQLObject but sqlobject/cache.py hasn't
really changed in 2 years, so this applies to current and past versions.
I operate a website that runs around 4million requests a day through
an application using SQLObject.
At about mid day I end up with large amount of weakref objects and
slowing response times.
Top 6 objects in memory:
2887448 <type 'weakref'>
209963 <type 'dict'>
68189 <type 'tuple'>
45584 <type 'weakproxy'>
45584 <class 'sqlobject.main.SQLObjectState'>
45584 <class 'sqlobject.declarative.sqlmeta'>
These objects are from the CacheFactory.expiredCache dictionary that
holds weak references to SQLObject instances by id. Some of these
dictionaries reach ~1.5 million keys, each SQLObject.get() request has
to look into this dictionary which results in an access slowdown and
memory growth. When the object behind the weak reference is collected,
a weakref with a dead object is left in the cache, In my case this is
a memory leak.
Any comments or improvements on this method are appreciated.
There are lots of ways to solve this problem but I think this is the
least disruptive. Weakref callbacks can't be used because the
"release" callback happens in the cull method.
I do 2 things in the cull method
1 - cull dead weak references out of the expiredCache.
2 - avoid placing dead weak references into the expiredCache
def cull(self):
"""
Runs through the cache and expires objects. E.g., if
``cullFraction`` is 3, then every third object is moved to
the 'expired' (aka weakref) cache.
"""
self.lock.acquire()
try:
#remove dead references from the expired cache
keys = self.expiredCache.keys()
for key in keys:
if self.expiredCache[key]() is None:
self.expiredCache.pop(key, None)
keys = self.cache.keys()
for i in xrange(self.cullOffset, len(keys),
self.cullFraction):
id = keys[i]
# create a weakref, then remove from the cache
obj = ref(self.cache[id])
del self.cache[id]
#the object may have been gc'd when removed from the
cache
#above, no need to place in expiredCache
if obj() is not None:
self.expiredCache[id] = obj
# This offset tries to balance out which objects we
# expire, so no object will just hang out in the cache
# forever.
self.cullOffset = (self.cullOffset + 1) % self.cullFraction
finally:
self.lock.release()
Thanks,
Jason
------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
sqlobject-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
