Re: httplib incredibly slow :-(

2009-09-07 Thread Chris Withers
Dieter Maurer wrote: Chris Withers ch...@simplistix.co.uk writes on Thu, 13 Aug 2009 08:20:37 +0100: ... I've already established that the file downloads in seconds with [something else], so I'd like to understand why python isn't doing the same and fix the problem... A profile might help to

Re: httplib incredibly slow :-(

2009-08-19 Thread Aahz
In article mailman.133.1250270175.2903.python-l...@python.org, Chris Withers ch...@simplistix.co.uk wrote: Aahz wrote: What do you need to know for a decent example? Simple download of a file from a url with some auth headers added would do me. Well, I've hacked up some sample code from my

Re: httplib incredibly slow :-(

2009-08-17 Thread Chris Withers
i3dmaster wrote: Just wanted to check if you can try turning on the debug mode for httplib and see if you can read a bit more debug info on where the calls get hung. In your example, it would be conn.set_debuglevel(1) I had a look through the code this debug level controls and I don't see any

Re: httplib incredibly slow :-(

2009-08-14 Thread Chris Withers
Aahz wrote: Sorry, I mostly have been working on our Mac port, so I'm not sure what's needed to make this work on Windows. Did you try downloading the PyCurl binary? Maybe it statically links libcurl on Windows. Shame it's not available as a bdist_egg, that's what I'm really after... What

Re: httplib incredibly slow :-(

2009-08-14 Thread Dieter Maurer
Chris Withers ch...@simplistix.co.uk writes on Thu, 13 Aug 2009 08:20:37 +0100: ... I've already established that the file downloads in seconds with [something else], so I'd like to understand why python isn't doing the same and fix the problem... A profile might help to understand what the

Re: httplib incredibly slow :-(

2009-08-13 Thread Chris Withers
David Robinow wrote: On Wed, Aug 12, 2009 at 12:37 PM, Chris Withersch...@simplistix.co.uk wrote: David Stanek wrote: Also on the same box where you run this script can you test with curl or wget? It's a Windows box, so no :-( Why not? http://users.ugent.be/~bpuype/wget/

Re: httplib incredibly slow :-(

2009-08-13 Thread David Robinow
On Thu, Aug 13, 2009 at 3:20 AM, Chris Withersch...@simplistix.co.uk wrote: David Robinow wrote: On Wed, Aug 12, 2009 at 12:37 PM, Chris Withersch...@simplistix.co.uk wrote: David Stanek wrote: Also on the same box where you run this script can you test with curl or wget? It's a Windows

Re: httplib incredibly slow :-(

2009-08-13 Thread Aahz
In article mailman.4613.1250033136.8015.python-l...@python.org, Chris Withers ch...@simplistix.co.uk wrote: Aahz wrote: In article mailman.4598.1250022343.8015.python-l...@python.org, Chris Withers ch...@simplistix.co.uk wrote: Does anyone know of an alternative library for creating http

Re: httplib incredibly slow :-(

2009-08-12 Thread Chris Withers
Answering myself... Chris Withers wrote: In article mailman.4598.1250022343.8015.python-l...@python.org, Chris Withers ch...@simplistix.co.uk wrote: Does anyone know of an alternative library for creating http requests and getting their responses that's faster but hopefully has a similar

Re: httplib incredibly slow :-(

2009-08-12 Thread Max Erickson
Chris Withers ch...@simplistix.co.uk wrote: I'm still reeling from what seems to be such a huge problem with httplib that seem to be largely ignored :-( Chris There is an httplib2 (but I don't know anything further about it...): http://code.google.com/p/httplib2/ Calling wget or curl

Re: httplib incredibly slow :-(

2009-08-12 Thread shaileshkumar
We use PyCURL on Windows. http://pycurl.sourceforge.net/ provides pre- built versions for Windows and it works out of the box. - Shailesh On Aug 12, 7:14 pm, Max Erickson maxerick...@gmail.com wrote: Chris Withers ch...@simplistix.co.uk wrote: I'm still reeling from what seems to be such a

Re: httplib incredibly slow :-(

2009-08-12 Thread David Stanek
On Tue, Aug 11, 2009 at 4:25 PM, Chris Withersch...@simplistix.co.uk wrote: Hi All, I'm using the following script to download a 150Mb file: from base64 import encodestring from httplib import HTTPConnection from datetime import datetime conn = HTTPSConnection('localhost') headers = {}

Re: httplib incredibly slow :-(

2009-08-12 Thread Chris Withers
Max Erickson wrote: There is an httplib2 (but I don't know anything further about it...): http://code.google.com/p/httplib2/ I had a look, it uses httplib, so will likely suffer from the same problems... Calling wget or curl using a subprocess is probably as easy as it is ugly, I use the

Re: httplib incredibly slow :-(

2009-08-12 Thread Shailesh Kumar
Yes it includes libcurl. I didn't have to install it separately. I still continue to use Python 2.4. So cannot say about Python 2.6. - Shailesh On Wed, Aug 12, 2009 at 10:23 PM, Chris Withers ch...@simplistix.co.ukwrote: shaileshkumar wrote: We use PyCURL on Windows.

Re: httplib incredibly slow :-(

2009-08-12 Thread Chris Withers
shaileshkumar wrote: We use PyCURL on Windows. http://pycurl.sourceforge.net/ provides pre- built versions for Windows and it works out of the box. Does it include libcurl? Are these builds available for Python 2.6? Chris -- Simplistix - Content Management, Batch Processing Python

Re: httplib incredibly slow :-(

2009-08-12 Thread Chris Withers
David Stanek wrote: I tried to reproduce this, but I could not. Could you paste in the output of your script? Not sure how that'll help, but sure: 2009-08-11 21:27:59.153000 request: 0:00:00.109000 response: 0:00:00.109000 read: 0:24:31.266000 Also on the same box where you run this script

Re: httplib incredibly slow :-(

2009-08-12 Thread i3dmaster
On Aug 12, 9:37 am, Chris Withers ch...@simplistix.co.uk wrote: David Stanek wrote: I tried to reproduce this, but I could not. Could you paste in the output of your script? Not sure how that'll help, but sure: 2009-08-11 21:27:59.153000 request: 0:00:00.109000 response: 0:00:00.109000

Re: httplib incredibly slow :-(

2009-08-12 Thread David Robinow
On Wed, Aug 12, 2009 at 12:37 PM, Chris Withersch...@simplistix.co.uk wrote: David Stanek wrote: Also on the same box where you run this script can you test with curl or wget? It's a Windows box, so no :-( Why not? http://users.ugent.be/~bpuype/wget/ http://curl.haxx.se/download.html --

httplib incredibly slow :-(

2009-08-11 Thread Chris Withers
Hi All, I'm using the following script to download a 150Mb file: from base64 import encodestring from httplib import HTTPConnection from datetime import datetime conn = HTTPSConnection('localhost') headers = {} auth = 'Basic '+encodestring('username:password').strip()

Re: httplib incredibly slow :-(

2009-08-11 Thread Aahz
In article mailman.4598.1250022343.8015.python-l...@python.org, Chris Withers ch...@simplistix.co.uk wrote: Does anyone know of an alternative library for creating http requests and getting their responses that's faster but hopefully has a similar interface? PyCurl -- Aahz

Re: httplib incredibly slow :-(

2009-08-11 Thread Chris Withers
Aahz wrote: In article mailman.4598.1250022343.8015.python-l...@python.org, Chris Withers ch...@simplistix.co.uk wrote: Does anyone know of an alternative library for creating http requests and getting their responses that's faster but hopefully has a similar interface? PyCurl This seems