I'm no SQLObject guru, but I would get somebody with some sql savvy to look
at your queries. I'm aware of 2 common tricks:
1. minimize joins and just querie each table separately and put them
together yourself.
2. Use a stored Querie the kind with "?" in it.
This advice came from an eBay DBA
If your doing queries rather than updates, you can quickly create SQLObject
"like" objects using the DictionaryCursor and then passing that to a generic
constructor like
class MyGenericObject:
def __init__(self, a_dict_from_the_query):
self.__dict__.update(a_dict_from_the_query)
I really like SQLObject, but if you're just doing queries (where you have
the bottleneck) it may be overkill.
-----Original Message-----
From: [email protected] [mailto:[EMAIL PROTECTED]
Behalf Of [EMAIL PROTECTED]
Sent: Saturday, December 29, 2007 9:08 AM
To: TurboGears
Subject: [TurboGears] Re: RAM use/gotchas in tg apps?
Hi Iain, recently I had some trouble with our showmedo.com and large
memory usage which we solved...
We're using TG 1.0.1 (feb 2007), SQLObject, Genshi. Most of the site
is rendered on-the-fly and we knew nothing about SQLObject's
behaviour, so our sql queries aren't terribly tight, nor did we have
much SQL experience beforehand.
Our biggest issue was having *lots* of SQLObject queries (e.g. 2000
for a page) and using python logic to filter the results, and this was
happening on e.g. the RSS feeds which are called frequently. This
caused big CPU spikes per call and consistently growing memory usage.
Our RSS feeds get pinged every 4 seconds. You don't have to remind me
just how bad this bit of code was :-)
Having solved these problems we tend to hover at 65mb RSS, previously
we'd reach 100mb every day and the process would be restarted by our
host. We generally get 8000 page views per day. I leave the process
up for a week inbetween site updates.
My guess is that the constant and frequent SQL queries caused lots of
objects to be added to the heap which were only slowly being cleared,
hence the continuous growth in memory usage. Having removed many of
the queries we now have stable memory usage.
I kept an eye on the garbage counter - import gc; print
len(gc.garbage) - and wired this into a stats page on the site, but we
never had any uncollected garbage. We did oscillate between 160k and
200k objects in the python instance (via len(gc.get_objects()), and
this was clearly growing whilst we had our 'leak'. Forcing the
garbage collector to collect didn't reduce the RSS.
My friend John gave me a hand debugging our memory usage, he wrote up
two blog entries based on some of our work which might help others:
http://psychicorigami.com/2007/10/27/a-little-sqlobject-performance-guide/
http://psychicorigami.com/2007/12/16/using-raw-sql-with-sqlobject-and-keepin
g-the-object-y-goodness/
HTH,
Ian.
ps. I don't track the email on this account...use ian AT showmedo DOT
com for any personal correspondence.
On Dec 24, 8:21 am, iain duncan <[EMAIL PROTECTED]> wrote:
> On Mon, 2007-24-12 at 00:19 -0800, iain duncan wrote:
>
> > Hi folks, I'm clearly doing something wrong because my tg apps get all
> > bloated after they have been running a while. I'm wondering if anyone
> > can comment on gotchas they encountered with their apps in that regard.
> > I'm using CherryPy session variables, but I believe they are getting
> > deleted properly ( I could well be wrong there, will be investigating
> > that ). I also have some widgets instantiated in static space. Other
> > than that I'm confused, because I *know* they aren't getting hammered
> > with hits. Deployment is apache mod_proxy to the app with supervisord
> > keeping it up.
>
> > One question, my superivor app was checking on the site by hitting a
> > method of the root controller once a minute. Could that be eating ram
> > from a keep-alive scenario?
>
> FWIW, the apps seem to start at about 20megs and then after 10 minutes
> or so get up to 90! And when I checked on it after it had been running
> for days, it was at 130 megs. :/
>
> Thanks
> Iain
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"TurboGears" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/turbogears?hl=en
-~----------~----~----~----~------~----~------~--~---
