Hi Iain, recently I had some trouble with our showmedo.com and large memory usage which we solved...
We're using TG 1.0.1 (feb 2007), SQLObject, Genshi. Most of the site is rendered on-the-fly and we knew nothing about SQLObject's behaviour, so our sql queries aren't terribly tight, nor did we have much SQL experience beforehand. Our biggest issue was having *lots* of SQLObject queries (e.g. 2000 for a page) and using python logic to filter the results, and this was happening on e.g. the RSS feeds which are called frequently. This caused big CPU spikes per call and consistently growing memory usage. Our RSS feeds get pinged every 4 seconds. You don't have to remind me just how bad this bit of code was :-) Having solved these problems we tend to hover at 65mb RSS, previously we'd reach 100mb every day and the process would be restarted by our host. We generally get 8000 page views per day. I leave the process up for a week inbetween site updates. My guess is that the constant and frequent SQL queries caused lots of objects to be added to the heap which were only slowly being cleared, hence the continuous growth in memory usage. Having removed many of the queries we now have stable memory usage. I kept an eye on the garbage counter - import gc; print len(gc.garbage) - and wired this into a stats page on the site, but we never had any uncollected garbage. We did oscillate between 160k and 200k objects in the python instance (via len(gc.get_objects()), and this was clearly growing whilst we had our 'leak'. Forcing the garbage collector to collect didn't reduce the RSS. My friend John gave me a hand debugging our memory usage, he wrote up two blog entries based on some of our work which might help others: http://psychicorigami.com/2007/10/27/a-little-sqlobject-performance-guide/ http://psychicorigami.com/2007/12/16/using-raw-sql-with-sqlobject-and-keeping-the-object-y-goodness/ HTH, Ian. ps. I don't track the email on this account...use ian AT showmedo DOT com for any personal correspondence. On Dec 24, 8:21 am, iain duncan <[EMAIL PROTECTED]> wrote: > On Mon, 2007-24-12 at 00:19 -0800, iain duncan wrote: > > > Hi folks, I'm clearly doing something wrong because my tg apps get all > > bloated after they have been running a while. I'm wondering if anyone > > can comment on gotchas they encountered with their apps in that regard. > > I'm using CherryPy session variables, but I believe they are getting > > deleted properly ( I could well be wrong there, will be investigating > > that ). I also have some widgets instantiated in static space. Other > > than that I'm confused, because I *know* they aren't getting hammered > > with hits. Deployment is apache mod_proxy to the app with supervisord > > keeping it up. > > > One question, my superivor app was checking on the site by hitting a > > method of the root controller once a minute. Could that be eating ram > > from a keep-alive scenario? > > FWIW, the apps seem to start at about 20megs and then after 10 minutes > or so get up to 90! And when I checked on it after it had been running > for days, it was at 130 megs. :/ > > Thanks > Iain --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "TurboGears" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/turbogears?hl=en -~----------~----~----~----~------~----~------~--~---
