Is there any real benefit to using WSGIAppController with DataApp over just returning the stringio_object.getvalue()
On Apr 25, 3:58 pm, Adrian <[email protected]> wrote: > I was able to fix the memory issue by using the (undocumented...) > WSGIAppController. I actually had to go to tg/controllers/ > wsgiappcontroller.py to figure what the hell was going on, but I came > up with this solution. > > After creating the tarfield in a StringIO object, I store > theTarFile.getvalue() in some variable (tmp), then close all of the > open StringIO objects. Instead of returning the raw bytestream, I do: > > return WSGIAppController(DataApp(tmp, content_type='application/x-tar; > charset=utf8', content_disposition='attachment; > filename="ImageRequest.tar"'))._default(kwargs) > > Now, the memory usage doesn't accumulate, and does indeed drop down to > a (rather bloated) baseline of ~700MB. > > I see there is a ticket to do so, but you guys should really document > WSGIAppController!!http://sourceforge.net/p/turbogears2/tickets/10/ > > On Apr 25, 1:53 pm, Adrian <[email protected]> wrote: > > > > > How long does it take to fall down to 600MB? I just did 10 downloads > > from the page and the process is still holding onto 1.2 GB of memory > > and has been that way for ~10 minutes. I'm running on Mac OS 10.6, but > > this same behavior happens on the Red Hat server I am mainly using. > > > I agree that when I comment out the "return" the memory usage drops > > down to some baseline, but obviously I need to serve the files and so > > I need to figure out why returning the data causes the memory to fill > > up. Do you have any suggestions? I just want to wipe the file out of > > memory after it is returned. > > > Thanks, > > Adrian > > > On Apr 25, 1:22 pm, "Diez B. Roggisch" <[email protected]> wrote: > > > > On Apr 25, 2011, at 5:41 PM, Adrian wrote: > > > > > Sorry for the wait, this weekend got a little crazy.. Here is a tar/ > > > > zipped Turbogears project I created:http://a-p-w.com/tarfile-test.tar.gz > > > > > I did the simplest possible thing "paster quickstart" and followed the > > > > directions. Then I added a function to 'root.py' called 'returnImages' > > > > that just tars up 3 files (that I included in the tarball) and returns > > > > the bytestream for the tar file. I just tried this on my machine - I > > > > downloaded the file several times and watched the memory usage climb > > > > accordingly. It's set up to run on localhost:8080, so the page to go > > > > to ishttp://localhost:8080/returnImages > > > > After fixing a missing import, I can run the example. > > > > But I don't see a leak. The memory usage is increasing, and nears 1GB at > > > times. > > > > But it always falls back to some base-line, around 600MB. > > > > I re-did the experiment with TG20. Two observations: > > > > - the memory baseline was roughly half of what TG21 created. > > > - the app seemed to run *much* faster > > > > I don't have explanations for either of the phenomena. > > > > On a probably related, but more general note: Just because Python > > > collects garbage, and might even return memory to the OS, the memory > > > consumption is not going down visibly often. The reason is that the OS > > > may well chose to assign the peak mem to the process, because it's > > > avoiding offset costs for memory page assignment and some funky other > > > stuff OSses do. > > > > Another observation: the memory consumption was massively reduced, when I > > > just didn't return the actual data. So there might in fact be some lazy > > > and eventually gc'ed data-structure responsible for this hi peaking > > > memory, and it might well be worth examining that. But it's not a leak. > > > > Diez -- You received this message because you are subscribed to the Google Groups "TurboGears" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/turbogears?hl=en.
