I was able to fix the memory issue by using the (undocumented...) WSGIAppController. I actually had to go to tg/controllers/ wsgiappcontroller.py to figure what the hell was going on, but I came up with this solution.
After creating the tarfield in a StringIO object, I store theTarFile.getvalue() in some variable (tmp), then close all of the open StringIO objects. Instead of returning the raw bytestream, I do: return WSGIAppController(DataApp(tmp, content_type='application/x-tar; charset=utf8', content_disposition='attachment; filename="ImageRequest.tar"'))._default(kwargs) Now, the memory usage doesn't accumulate, and does indeed drop down to a (rather bloated) baseline of ~700MB. I see there is a ticket to do so, but you guys should really document WSGIAppController!! http://sourceforge.net/p/turbogears2/tickets/10/ On Apr 25, 1:53 pm, Adrian <[email protected]> wrote: > How long does it take to fall down to 600MB? I just did 10 downloads > from the page and the process is still holding onto 1.2 GB of memory > and has been that way for ~10 minutes. I'm running on Mac OS 10.6, but > this same behavior happens on the Red Hat server I am mainly using. > > I agree that when I comment out the "return" the memory usage drops > down to some baseline, but obviously I need to serve the files and so > I need to figure out why returning the data causes the memory to fill > up. Do you have any suggestions? I just want to wipe the file out of > memory after it is returned. > > Thanks, > Adrian > > On Apr 25, 1:22 pm, "Diez B. Roggisch" <[email protected]> wrote: > > > > > On Apr 25, 2011, at 5:41 PM, Adrian wrote: > > > > Sorry for the wait, this weekend got a little crazy.. Here is a tar/ > > > zipped Turbogears project I created:http://a-p-w.com/tarfile-test.tar.gz > > > > I did the simplest possible thing "paster quickstart" and followed the > > > directions. Then I added a function to 'root.py' called 'returnImages' > > > that just tars up 3 files (that I included in the tarball) and returns > > > the bytestream for the tar file. I just tried this on my machine - I > > > downloaded the file several times and watched the memory usage climb > > > accordingly. It's set up to run on localhost:8080, so the page to go > > > to ishttp://localhost:8080/returnImages > > > After fixing a missing import, I can run the example. > > > But I don't see a leak. The memory usage is increasing, and nears 1GB at > > times. > > > But it always falls back to some base-line, around 600MB. > > > I re-did the experiment with TG20. Two observations: > > > - the memory baseline was roughly half of what TG21 created. > > - the app seemed to run *much* faster > > > I don't have explanations for either of the phenomena. > > > On a probably related, but more general note: Just because Python collects > > garbage, and might even return memory to the OS, the memory consumption is > > not going down visibly often. The reason is that the OS may well chose to > > assign the peak mem to the process, because it's avoiding offset costs for > > memory page assignment and some funky other stuff OSses do. > > > Another observation: the memory consumption was massively reduced, when I > > just didn't return the actual data. So there might in fact be some lazy and > > eventually gc'ed data-structure responsible for this hi peaking memory, and > > it might well be worth examining that. But it's not a leak. > > > Diez -- You received this message because you are subscribed to the Google Groups "TurboGears" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/turbogears?hl=en.
