On Sat, 15 Dec 2012 06:17:19 +1100 Ben Leslie <be...@benno.id.au> wrote: > The http.client HTTPConnection._send_output method has an optimization for > avoiding bad interactions between delayed-ack and the Nagle algorithm: > > http://hg.python.org/cpython/file/f32f67d26035/Lib/http/client.py#l884 > > Unfortunately this interacts rather poorly if the case where the > message_body is a bytes instance and is rather large. > > If the message_body is bytes it is appended to the headers, which causes a > copy of the data. When message_body is large this duplication of data can > cause a significant spike in memory usage. > > (In my particular case I was uploading a 200MB file to 30 hosts at the same > leading to memory spikes over 6GB. > > I've solved this by subclassing and removing the optimization, however I'd > appreciate thoughts on how this could best be solved in the library itself. > > Options I have thought of are: > > 1: Have some size threshold on the copy. A little bit too much magic. > Unclear what the size threshold should be.
I think a hardcoded threshold is the right thing to do. It doesn't sound very useful to try doing a single send() call when you have a large chunk of data (say, more than 1 MB). Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
