On 07:02 pm, [email protected] wrote: >On 10-12-21 12:47 PM, [email protected] wrote: >>On 04:43 pm, [email protected] wrote: >>> >>>OK, I guess I'm being slow :-( Here's another version, same results. >>> >>>import os >>>from twisted.internet import reactor >>>from twisted.web.server import Site >>>from twisted.web.resource import Resource >>> >>>CHUNK_SIZE = 32*1024 >>>data = os.urandom(10*1024*1024) >>>chunks = [] >>> >>>def make_chunks(): >>> s = 0 >>> for chunk in iter(lambda: data[s:s+CHUNK_SIZE], ''): >>> chunks.append(chunk) >>> s = s + CHUNK_SIZE >>> >>>class TestPage(Resource): >>> isLeaf = True >>> >>> def render_GET(self, request): >>> for chunk in chunks: >>> request.write(chunk) >>> >>>make_chunks() >>>root = Resource() >>>root.putChild('test', TestPage()) >>>reactor.listenTCP(8880, Site(root)) >>>reactor.run() >> >>This version has flat memory usage on my system - 33MB all the way >>through. >>Jean-Paul > >Did you try this running the clients on a different machine than the >one >running the server? I find that if I run both the server and the >clients >(using httperf) on the same machine things behave differently. I can >only see the memory consumption problem consistently when running the >clients on a separate machine (which would be my real use case). > >BTW, I'm running these test on Ubuntu 10.04, Python 2.6.5, twisted >10.0.0-2ubuntu2.
Argh. I just ran it over loopback, so I wasn't accurately testing the case you described. James Knight just reminded me that the transport will still find a way to copy data in this case, so you're right that this still isn't a working solution. Instead, you have to go all the way to producers/consumers, and only write more data to the transport buffer when it has finished dealing with what you previously gave it. Request implements IConsumer, which means you can pass an IProducer provider to its registerProducer method. When the send buffer gets close to empty, the IProducer's resumeProducing method will be called and you can write some more bytes to the Request. There are some more complications and subtleties in the IProducer/IConsumer interfaces, which you can read about at <http://twistedmatrix.com/documents/current/core/howto/producers.html>. Basically, you'll end up with something like: from twisted.python.log import err from twisted.protocols.basic import FileSender from twisted.web.server import NOT_DONE_YET def render_GET(self, request): producer = FileSender() d = producer.beginFileTransfer(StringIO(chunk), request) def finished(ignored): request.finish() d.addErrback(err, "Streaming data to client failed") d.addCallback(finished) return NOT_DONE_YET Sorry for the earlier mis-information. Jean-Paul _______________________________________________ Twisted-web mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
