On 10-12-21 02:24 PM, [email protected] wrote: > On 07:02 pm, [email protected] wrote: >> On 10-12-21 12:47 PM, [email protected] wrote: >>> On 04:43 pm, [email protected] wrote: >>>> >>>> OK, I guess I'm being slow :-( Here's another version, same results. >>>> >>>> import os >>> >from twisted.internet import reactor >>> >from twisted.web.server import Site >>> >from twisted.web.resource import Resource >>>> >>>> CHUNK_SIZE = 32*1024 >>>> data = os.urandom(10*1024*1024) >>>> chunks = [] >>>> >>>> def make_chunks(): >>>> s = 0 >>>> for chunk in iter(lambda: data[s:s+CHUNK_SIZE], ''): >>>> chunks.append(chunk) >>>> s = s + CHUNK_SIZE >>>> >>>> class TestPage(Resource): >>>> isLeaf = True >>>> >>>> def render_GET(self, request): >>>> for chunk in chunks: >>>> request.write(chunk) >>>> >>>> make_chunks() >>>> root = Resource() >>>> root.putChild('test', TestPage()) >>>> reactor.listenTCP(8880, Site(root)) >>>> reactor.run() >>> >>> This version has flat memory usage on my system - 33MB all the way >>> through. >>> Jean-Paul >> >> Did you try this running the clients on a different machine than the >> one >> running the server? I find that if I run both the server and the >> clients >> (using httperf) on the same machine things behave differently. I can >> only see the memory consumption problem consistently when running the >> clients on a separate machine (which would be my real use case). >> >> BTW, I'm running these test on Ubuntu 10.04, Python 2.6.5, twisted >> 10.0.0-2ubuntu2. > > Argh. I just ran it over loopback, so I wasn't accurately testing the > case you described. James Knight just reminded me that the transport > will still find a way to copy data in this case, so you're right that > this still isn't a working solution. Instead, you have to go all the > way to producers/consumers, and only write more data to the transport > buffer when it has finished dealing with what you previously gave it. > > Request implements IConsumer, which means you can pass an IProducer > provider to its registerProducer method. When the send buffer gets > close to empty, the IProducer's resumeProducing method will be called > and you can write some more bytes to the Request. > > There are some more complications and subtleties in the > IProducer/IConsumer interfaces, which you can read about at > <http://twistedmatrix.com/documents/current/core/howto/producers.html>. > > Basically, you'll end up with something like: > > from twisted.python.log import err > from twisted.protocols.basic import FileSender > from twisted.web.server import NOT_DONE_YET > > def render_GET(self, request): > producer = FileSender() > d = producer.beginFileTransfer(StringIO(chunk), request) > def finished(ignored): > request.finish() > d.addErrback(err, "Streaming data to client failed") > d.addCallback(finished) > return NOT_DONE_YET > > Sorry for the earlier mis-information. > > Jean-Paul > > _______________________________________________ > Twisted-web mailing list > [email protected] > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Thanks again Jean-Paul, This one will take me some time to digest and prototype. I'll give it a whirl to try to implement a solution and I'll post back my results. -- Pedro _______________________________________________ Twisted-web mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
