On Fri, Sep 4, 2009 at 4:28 AM, Simon Cross<hodgestar+python...@gmail.com> wrote: > On Fri, Sep 4, 2009 at 1:11 PM, Chris Withers<ch...@simplistix.co.uk> wrote: >> Am I right in reading this as most of the time is being spent in httplib's >> HTTPResponse._read_chunked and none of the methods it calls? >> >> If so, is there a better way that a bunch of print statements to find where >> in that method the time is being spent? > > Well, since the source for _read_chunked includes the comment > > # XXX This accumulates chunks by repeated string concatenation, > # which is not efficient as the number or size of chunks gets big. > > you might gain some speed improvement with minimal effort by gathering > the read data chunks into a list and then returning "".join(chunks) at > the end.
+1 on trying this. Constructing a 116MB string by concatenating 1KB buffers surely must take forever. (116MB divided by 85125 recv() calls give 1365 byte per chunk, which is awful.) The HTTP/1.0 business looks like a red herring. Also agreed that this is an embarrassment. -- --Guido van Rossum (home page: http://www.python.org/~guido/) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
