Oleg Broytman wrote, On 27/08/2012 10:57: > Hello! > > On Mon, Aug 27, 2012 at 10:32:50AM +0200, Gert Burger <gertbur...@gmail.com> > wrote: >> We have a RPC server which CPU usage increases over time, after a week >> it will max out one core. >> >> >From my testing(Using pyrasite's dump stack) it seems that all the time >> is spend in: >> >> File >> "/usr/local/lib/python2.6/dist-packages/SQLObject-1.3.1-py2.6.egg/sqlobject/cache.py", >> line 263, in allIDs >> for id, value in self.expiredCache.items(): >> >> Currently the overhead for each commit and rollback is around 2-3 seconds. >> >> Almost all of our tables have caching disabled via 'sqlmeta: cacheValues >> = False'. > > sqlmeta.cacheValues disables caching of attributes -- with > cacheValues=False every time a program reads and attribute > (value = row.column) SQLObject issues a SELECT query. > There is also row caching -- SQLObject caches rows in the connection > cache. The code above is related to that cache. To disable the cache > pass cache=False to connection constructor or in the DB URI: > schema://host:port/db?cache=0.
I have this already. > >> No state is kept in memory between requests and we typically write >> significantly more rows to the DB than what is read, ie. A lot of writes >> which are not read often and a few hundred rows which are read often and >> that can be changed by other processes. > > So you need mass insertion. See the related FAQ entry: > http://sqlobject.org/FAQ.html#how-to-do-mass-insertion > > Oleg. > Most of our inserts occur in small batches which wont benefit much from bypassing SQLO. From my profiling it doesn't seem like that is the problem. It seems like the expiredCache for each table is growing continuously. Since my last email the size of expiredCache for some tables have grown to over a million entries and it never seems to decrease. In total the expired caches have about 4 million entries over all tables and current memory usage(150MB RES) doesn't match that amount of objects(of row data). So my next question is, how do I prevent the expired caches from growing uncontrollably? For some reason the weak references are staying active and therefore they are not removed from the caches. Regards Gert Burger ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
