Re: urllib2 urlopen takes too much time

2017-12-08 Thread Python 
On Sun, Dec 03, 2017 at 09:49:45AM -0800, cpol...@vt.edu wrote:
> On Tuesday, June 23, 2009 at 11:09:30 PM UTC-4, Aahz wrote:
> > In article ,
> > =?UTF-8?Q?Filip_Gruszczy=C5=84ski?=   wrote:
> > >
> > >I have encountered a performance problem using suds, which was traced
> > >down to _socket.recv. I am calling some web services and each of them
> > >uses about 0.2 sec and 99% of this time is spent on urllib2.urlopen,
> > >while the rest of the call is finished in milliseconds.
> >
> > What happens if you use urlopen() by itself?
>
> I am having the same issue.  Did you ever figure out a solution?
> The time delay is also the same, from .2 to .25 sec.  I need to stay
> with urlopen though so I can do a custom request header

Depending on your network conditions, this doesn't seem all that outrageous. 
Using tcpdump or some other packet sniffer should reveal what is going on, but
most likely urlopen() needs to do all of the following, every time it is
called:

  - resolve the address via DNS
 + 1 round trip: DNS request + response; possibly more if DNS
   doesn't have host entry and forwards...
  - initiate a TCP connection
 + three way handshake = 1.5 round trip times
  - send the request (.5 round trip)
  - wait for server processing (unknown time)
  - receive the response (.5 round trip time)

If the DNS server is busy, or the SOAP server is busy, add that latency to the
mix.  Assuming NO latency due to server load, you're still looking at 3.5 round
 trips to service the request, or 7 one-way (possibly 6, if the ACK contains
the request, which IIRC is allowed). .2s / 7 = ~29ms, so if the latency between
 server and client is 29ms or higher, that explains everything.  On a LAN I'd
hope for much better than 29ms latency (1ms is more typical), but it's not
outside the realm of possibility depending on network architecture and load.
Ping to the server will tell you that... if your network isn't blocking ICMP.

If the C# app is faster, my guess would be it is caching the DNS response
and/or maintaining persistent connections to the server, whereas urllib isn't,
though that's just a guess... I'm not familiar with the implementations of
either.  But again, a packet sniffer should show you exactly what's happening
and who is being slow.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: urllib2 urlopen takes too much time

2017-12-03 Thread Python 
On Sun, Dec 03, 2017 at 09:49:45AM -0800, cpol...@vt.edu wrote:
> On Tuesday, June 23, 2009 at 11:09:30 PM UTC-4, Aahz wrote:
> > In article ,
> > =?UTF-8?Q?Filip_Gruszczy=C5=84ski?=   wrote:
> > >
> > >I have encountered a performance problem using suds, which was traced
> > >down to _socket.recv. I am calling some web services and each of them
> > >uses about 0.2 sec and 99% of this time is spent on urllib2.urlopen,
> > >while the rest of the call is finished in milliseconds. 
> > 
> > What happens if you use urlopen() by itself?
> 
> I am having the same issue.  Did you ever figure out a solution?
> The time delay is also the same, from .2 to .25 sec.  I need to stay
> with urlopen though so I can do a custom request header

Depending on your network conditions, this doesn't seem all that
outrageous.  Using tcpdump or some other packet sniffer should reveal
what is going on, but most likely urlopen() needs to do all of the
following, every time it is called:

  - resolve the address via DNS
 + 1 round trip: DNS request + response; possibly more if DNS
   doesn't have host entry and forwards...
  - initiate a TCP connection
 + three way handshake = 1.5 round trip times
  - send the request (.5 round trip)
  - wait for server processing (unknown time)
  - receive the response (.5 round trip time)

If the DNS server is busy, or the SOAP server is busy, add that
latency to the mix.  Assuming NO latency due to server load, you're
still looking at 3.5 round trips to service the request, or 7 one-way
(possibly 6, if the ACK contains the request, which IIRC is allowed).
.2s / 7 = ~29ms, so if the latency between server and client is 29ms
or higher, that explains everything.  On a LAN I'd hope for much
better than 29ms latency (1ms is more typical), but it's not outside
the realm of possibility depending on network architecture and load.
Ping to the server will tell you that... if your network isn't
blocking ICMP.

If the C# app is faster, my guess would be it is caching the DNS
response and/or maintaining persistent connections to the server,
whereas urllib isn't, though that's just a guess... I'm not familiar
with the implementations of either.  But again, a packet sniffer
should show you exactly what's happening and who is being slow. 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: urllib2 urlopen takes too much time

2017-12-03 Thread cpollio 
On Tuesday, June 23, 2009 at 11:09:30 PM UTC-4, Aahz wrote:
> In article ,
> =?UTF-8?Q?Filip_Gruszczy=C5=84ski?=   wrote:
> >
> >I have encountered a performance problem using suds, which was traced
> >down to _socket.recv. I am calling some web services and each of them
> >uses about 0.2 sec and 99% of this time is spent on urllib2.urlopen,
> >while the rest of the call is finished in milliseconds. 
> 
> What happens if you use urlopen() by itself?
> -- 
> Aahz (a...@pythoncraft.com)   <*> http://www.pythoncraft.com/
> 
> "as long as we like the same operating system, things are cool." --piranha

I am having the same issue.  Did you ever figure out a solution?  The time 
delay is also the same, from .2 to .25 sec.  I need to stay with urlopen though 
so I can do a custom request header
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: urllib2 urlopen takes too much time

2009-06-23 Thread Aahz 
In article mailman.1916.1245600033.8015.python-l...@python.org,
=?UTF-8?Q?Filip_Gruszczy=C5=84ski?=  grusz...@gmail.com wrote:

I have encountered a performance problem using suds, which was traced
down to _socket.recv. I am calling some web services and each of them
uses about 0.2 sec and 99% of this time is spent on urllib2.urlopen,
while the rest of the call is finished in milliseconds. 

What happens if you use urlopen() by itself?
-- 
Aahz (a...@pythoncraft.com)   * http://www.pythoncraft.com/

as long as we like the same operating system, things are cool. --piranha
-- 
http://mail.python.org/mailman/listinfo/python-list