[issue29533] urllib2 works slowly with proxy on windows

Change by Eryk Sun : -- versions: +Python 3.10, Python 3.8, Python 3.9 -Python 2.7, Python 3.6 ___ Python tracker ___ ___

[issue29533] urllib2 works slowly with proxy on windows

Marc Schlaich added the comment: Well, you can read the proxy settings from registry and write them to os.environ (no_proxy needs to be transformed as it has a different format). This will only take effect for the current process. -- ___ Python

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: I'm not sure that users of my program will like if I define such variables in their systems -- ___ Python tracker ___

[issue29533] urllib2 works slowly with proxy on windows

Marc Schlaich added the comment: BTW, you can workaround this issue by defining the `http_proxy` and `no_proxy` environment variables. In this case urllib isn't doing any DNS request. -- ___ Python tracker

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: I compared the behaviour of urllib with these browsers: Firefox("use system proxy" selected), Google Chrome, Yandex. And also Skype (requests to login.live.com). All of them are not doing DNS requests for proxy bypass handling as Marc expects. The result is

[issue29533] urllib2 works slowly with proxy on windows

Marc Schlaich added the comment: Julia, could you please add other major browsers/HTTP clients (Firefox, Chrome, curl, ...) to your comparison (compare_ie_urllib.txt). I would expect that Python/urllib is the only implementation doing DNS requests for proxy bypass handling. Please note that

[issue29533] urllib2 works slowly with proxy on windows

Marc Schlaich added the comment: This could be even a security issue. People might rely on a proxy as a privacy feature. In this case the proxy should do forward/reverse DNS requests and not the client. Doing DNS lookups to check for proxy bypass doesn't seem right. I don't think that major

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: Steve, do you mean that there should be no address to IE configuration from urllib? I could undertake it if I understand the task. gethostbyaddr() is ok. It just makes a reverse lookup, that some dns-servers work up too slow. The command "nslookup" also works

[issue29533] urllib2 works slowly with proxy on windows

Steve Dower added the comment: I think the point is that we don't want to be grabbing settings like this from other configuration locations. Ideally, there'd be a way to provide a list of "don't bypass the proxy for these names", which a caller could then read from the IE configuration if

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: Ok, but may be there are some Windows users, that have different opinion, who prefer to put up with this bug for the benefit of better performance. Could you leave them an opportunity to refuse this behavior of urllib? --

[issue29533] urllib2 works slowly with proxy on windows

Paul Moore added the comment: The behaviour you're describing for IE sounds like a bug to me. If you specify a host that should bypass the proxy, then that's what should happen - it shouldn't matter if you specify the host by IP address or by name. I'm -1 on Python trying to match IE bugs.

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: May be I described the problem not clearly enough, because English is not my native language, so I try to explain once again. In Windows there is an option "Do not use proxy server for address beginning with". I call this option . This option is invented for

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: Could someone look into my PR, please... -- ___ Python tracker ___ ___

[issue29533] urllib2 works slowly with proxy on windows

Changes by Julia Dolgova : -- pull_requests: +212 ___ Python tracker ___ ___ Python-bugs-list

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: I added variable smart_proxy_bypass into request module. If it's False then checking IP and FQDN of the host is skipped in proxy_bypass_registry function. I set the default value to False, because I think it's better when the behaviour of urllib corresponds

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: http://bugs.python.org/issue23384 - same problem -- ___ Python tracker ___

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: Why not to take it into account? Imagine that someone wants that requests to "ovinnik.canonical.com" should bypass proxy and requests to "ubuntu.com" souldn't. I don't know what for, it's just an assumption. He adds a hostname "ovinnik.canonical.com" into

[issue29533] urllib2 works slowly with proxy on windows

Steve Dower added the comment: My guess is that IE is implemented using lower level APIs and it can choose whether to bypass based on its own list. There's no reason for any other software to take its settings into account. That said, it would be great if urllib can avoid adding long delays,

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: I compared the behavior of IE and urllib. I put different addresses to the ("Do not use proxy server for address beginning with" setting), made different requests through IE and urllib and watched if the proxy was bypassed. IE doesn't even make a forward dns

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: The issue applies to 3.6 as well. I agree that the replacement of gethostbyaddr with gethostbyname_ex is not a solution. But is there a way to check whether a hostname is in the that doesn't bring to the reverse lookup? I suppose that IE doesn't make a

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: I sincerely appreciate your time. Thank you very much for your answer. I'll try to test this on python 3.5 -- ___ Python tracker

[issue29533] urllib2 works slowly with proxy on windows

Eryk Sun added the comment: gethostbyname_ex won't do a reverse lookup on an IP to get the fully-qualified domain name, which seems pointless for a function named getfqdn. I think calling gethostbyaddr is intentional here and goes back to the Python 1.x days. Also, FYI, socket_gethostbyaddr

[issue29533] urllib2 works slowly with proxy on windows

Steve Dower added the comment: There's a few reasons why you haven't heard a reply. First among them is that we're all volunteers with limited free time, and second is that we just migrated to github and all that free time is being consumed right now. Python 2.7 is only receiving security

[issue29533] urllib2 works slowly with proxy on windows

Julia Dolgova added the comment: Surely noone is concerned that programms written on python could work better when addressing to "python.org"? -- ___ Python tracker

[issue29533] urllib2 works slowly with proxy on windows

Changes by Julia Dolgova : -- keywords: +patch Added file: http://bugs.python.org/file46632/socket.patch ___ Python tracker ___

[issue29533] urllib2 works slowly with proxy on windows

Changes by Julia Dolgova : Added file: http://bugs.python.org/file46630/test.py ___ Python tracker ___

[issue29533] urllib2 works slowly with proxy on windows

Changes by Julia Dolgova : Added file: http://bugs.python.org/file46631/log.txt ___ Python tracker ___

[issue29533] urllib2 works slowly with proxy on windows

Changes by Julia Dolgova : Added file: http://bugs.python.org/file46629/socket.py ___ Python tracker ___

[issue29533] urllib2 works slowly with proxy on windows

New submission from Julia Dolgova: I've found that urllib works sometimes slowly on windows with proxy. To reproduce the issue: on Windows: 1. Turn on the option "use proxy" in "browser settings" in "control panel". No real proxy needed. The problem will come out before addressing to proxy.