Hi Fred. John (above) is just such friend with sql savvy :-) He's helped me remove a whole lot of the queries, with more clean-up to come. This clean-up has been responsible for lowering our RSS for sure.
I meant to also mention that our other site (services.showmedo.com) is a very simple TG site - that runs for weeks at a time at 20mb RSS between updates. It has basic logins, some graphing, some external url callbacks for logging - with 10s-100s of visitors per day. Ian. On Jan 2, 8:37 pm, "Fred Sells" <[EMAIL PROTECTED]> wrote: > I'm no SQLObject guru, but I would get somebody with some sql savvy to look > at your queries. I'm aware of 2 common tricks: > 1. minimize joins and just querie each table separately and put them > together yourself. > 2. Use a stored Querie the kind with "?" in it. > This advice came from an eBay DBA > > If your doing queries rather than updates, you can quickly create SQLObject > "like" objects using the DictionaryCursor and then passing that to a generic > constructor like > > class MyGenericObject: > def __init__(self, a_dict_from_the_query): > self.__dict__.update(a_dict_from_the_query) > > I really like SQLObject, but if you're just doing queries (where you have > the bottleneck) it may be overkill. > > -----Original Message----- > From: [email protected] [mailto:[EMAIL PROTECTED] > > Behalf Of [EMAIL PROTECTED] > Sent: Saturday, December 29, 2007 9:08 AM > To: TurboGears > Subject: [TurboGears] Re: RAM use/gotchas in tg apps? > > Hi Iain, recently I had some trouble with our showmedo.com and large > memory usage which we solved... > > We're using TG 1.0.1 (feb 2007), SQLObject, Genshi. Most of the site > is rendered on-the-fly and we knew nothing about SQLObject's > behaviour, so our sql queries aren't terribly tight, nor did we have > much SQL experience beforehand. > > Our biggest issue was having *lots* of SQLObject queries (e.g. 2000 > for a page) and using python logic to filter the results, and this was > happening on e.g. the RSS feeds which are called frequently. This > caused big CPU spikes per call and consistently growing memory usage. > Our RSS feeds get pinged every 4 seconds. You don't have to remind me > just how bad this bit of code was :-) > > Having solved these problems we tend to hover at 65mb RSS, previously > we'd reach 100mb every day and the process would be restarted by our > host. We generally get 8000 page views per day. I leave the process > up for a week inbetween site updates. > > My guess is that the constant and frequent SQL queries caused lots of > objects to be added to the heap which were only slowly being cleared, > hence the continuous growth in memory usage. Having removed many of > the queries we now have stable memory usage. > > I kept an eye on the garbage counter - import gc; print > len(gc.garbage) - and wired this into a stats page on the site, but we > never had any uncollected garbage. We did oscillate between 160k and > 200k objects in the python instance (via len(gc.get_objects()), and > this was clearly growing whilst we had our 'leak'. Forcing the > garbage collector to collect didn't reduce the RSS. > > My friend John gave me a hand debugging our memory usage, he wrote up > two blog entries based on some of our work which might help > others:http://psychicorigami.com/2007/10/27/a-little-sqlobject-performance-g...http://psychicorigami.com/2007/12/16/using-raw-sql-with-sqlobject-and... > g-the-object-y-goodness/ > > HTH, > Ian. > ps. I don't track the email on this account...use ian AT showmedo DOT > com for any personal correspondence. > > On Dec 24, 8:21 am, iain duncan <[EMAIL PROTECTED]> wrote: > > On Mon, 2007-24-12 at 00:19 -0800, iain duncan wrote: > > > > Hi folks, I'm clearly doing something wrong because my tg apps get all > > > bloated after they have been running a while. I'm wondering if anyone > > > can comment on gotchas they encountered with their apps in that regard. > > > I'm using CherryPy session variables, but I believe they are getting > > > deleted properly ( I could well be wrong there, will be investigating > > > that ). I also have some widgets instantiated in static space. Other > > > than that I'm confused, because I *know* they aren't getting hammered > > > with hits. Deployment is apache mod_proxy to the app with supervisord > > > keeping it up. > > > > One question, my superivor app was checking on the site by hitting a > > > method of the root controller once a minute. Could that be eating ram > > > from a keep-alive scenario? > > > FWIW, the apps seem to start at about 20megs and then after 10 minutes > > or so get up to 90! And when I checked on it after it had been running > > for days, it was at 130 megs. :/ > > > Thanks > > Iain --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "TurboGears" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/turbogears?hl=en -~----------~----~----~----~------~----~------~--~---
