On 02:34 pm, [email protected] wrote: >On 10-12-21 09:00 AM, [email protected] wrote: >>On 05:19 am, [email protected] wrote: >>>Hello, >>> >>>Here's a demo HTTP server that returns 10 MB of random data each time >>>a >>>client connects. >>> >>>import os >>>from twisted.internet import reactor >>>from twisted.web.server import Site >>>from twisted.web.resource import Resource >>> >>>data = os.urandom(10*1024*1024) >>> >>>class TestPage(Resource): >>> isLeaf = True >>> def render_GET(self, request): >>> return data >>> >>>root = Resource() >>>root.putChild('test', TestPage()) >>>reactor.listenTCP(8880, Site(root)) >>>reactor.run() >>> >>>Now, when I run N clients simultaneously from a different host I see >>>that the server's memory consumption increases by N*10 MB. I can't >>>reproduce this example when running the clients from the same host as >>>the server; the test goes so fast that I can't gather any useful >>>data. >>> >>>I run the test using the following httperf command on a different >>>host >>>and looking at the Gnome system monitor in the server (top will do as >>>well). >>> >>>httperf --server 192.168.1.10 --port 8880 --uri /test \ >>> --rate 10 --num-conn 10 >>> >>>When the server is idle memory consumption is 17.1 MB, but during the >>>test it jumps to 117.2 MB. My questions are then: >>> >>>1. Given that 'data' is a global variable, eventually read-only as >>>well, >>>why is it replicated for each request? And who is replicating it? >> >>It's copied as part of the process of writing it to the socket. You >>can't write 10MB at once, and you can't slice a string (to throw away >>the part that you did manage to write) without making a copy of part >>of >>it. >>>2, What would be the proper way to re-write this example so that >>>there >>>is one and only one 'data' structure at any time? >> >>Split data up into ~32kB-64kB chunks and write them to the request >>individually. Then each chunk can just be dropped with no copying. >> >>Jean-Paul >> >>_______________________________________________ >>Twisted-web mailing list >>[email protected] >>http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web > >Thanks Jean-Paul, > >Here is my modified example, unfortunately with the same bad results >regarding memory consumption. > >import os >from twisted.internet import reactor >from twisted.web.server import Site >from twisted.web.resource import Resource > >CHUNK_SIZE = 32*1024 >data = os.urandom(10*1024*1024) > >class TestPage(Resource): > isLeaf = True > > def render_GET(self, request): > s = 0 > for chunk in iter(lambda: data[s:s+CHUNK_SIZE], ''): > request.write(chunk) > s = s + CHUNK_SIZE
You've just moved the copying-by-slicing out of the transport and into your Resource. :) You need to do all that copying/slicing at the beginning, where it only needs to happen once, so that every render can share those allocated strings. Jean-Paul _______________________________________________ Twisted-web mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
