[issue7291] urllib2 cannot handle https with proxy requiring auth

[Alexey Namyotkin]

Alexey Namyotkin  added the comment:

It has been 5 years, now the urllib3 is actively used, but it also inherited 
this problem: if no authentication data has been received, then the method 
_tunnel raises an exception OSError that does not contain response headers. 
Accordingly, this exception cannot be handled. And perhaps this is an obstacle 
to building a convenient system of authentication on a proxy server in a widely 
used library requests (it would be nice to be able to just provide an argument 
proxy_auth, similar to how it is done for server authorization). Now, if a user 
wants to send a https request through a proxy that requires complex 
authentication (Kerberos, NTLM, Digest, other) using the urllib3, he must first 
send a separate request to the proxy, receive a response, extract the necessary 
data to form the header Proxy-Authorization, then generate this header and pass 
it to the ProxyManager. And if we are talking about Requests, then the 
situation there is worse, because you cannot pass proxy headers directly 
 (https://github.com/psf/requests/issues/2708).
If we were to aim to simplify the authentication procedure on the proxy server 
for the user, then where would we start, do we need to change the http.client 
so that the error returned by the method _tunnel contains headers? Or maybe 
it's not worth changing anything at all and the path with preliminary 
preparation by user of the header Proxy-Authorization is the only correct one? 
Martin Panter, could you also give your opinion? Thank you in advance.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Jessica Ridgley]

Change by Jessica Ridgley :


--
nosy: +Jessica Ridgley
versions: +Python 3.10, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Alexey Namyotkin]

Change by Alexey Namyotkin :


--
nosy: +alexey.namyotkin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Chi Hsuan Yen]

Chi Hsuan Yen added the comment:

Modify target versions to bugfix and feature branches

--
versions: +Python 3.7 -Python 2.6, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Chi Hsuan Yen]

Changes by Chi Hsuan Yen :


--
nosy: +Chi Hsuan Yen

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Martin Panter]

Martin Panter added the comment:

For the record, a while ago I think I made a patch implementing my 
HTTPConnection.detach() proposal. I can probably dig it up if anyone is 
interested.

However I gave up on fixing this bug in “urllib.request”. As far as I 
understand it, the framework does not distinguish the 407 Proxy Authentication 
Required error of the initial proxy CONNECT request from any potential 407 
response from a tunnelled connection. Perhaps a special case could be made; I 
think there are already lots of special cases. But the maze of urlopen() 
handlers is already too complicated and I decided this was too hard to bother 
working on. Sorry :)

--
stage:  - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Martin Panter]

Martin Panter added the comment:

I believe this also affects Python 3; see Issue 24333.

I think making the CONNECT response object available to the caller is the right 
general approach. But I really dislike raising an exception that holds a socket 
connection to be closed. (I know this is already done with 
urllib.error.HTTPError; let’s not repeat this in the “http.client” module!)

Ideally, at the “http.client” level I would prefer to avoid all the special 
set_tunnel() calls, and use the usual request() and getresponse() API to make 
the CONNECT request. This way the response status and header fields would be 
available just like any other response. For this approach to work, we would 
probably need to add a new HTTPConnection.detach() method that released the 
original socket reader and writer, and add a way to create a new HTTPConnection 
instance using these socket objects. This enhancement probably wouldn’t be 
appropriate for Python 2 or a bug fix release. But it seems the cleanest 
approach to me, and may also allow using HTTPConnection with the Upgrade header 
(e.g. for opportunistic encryption, HTTP 2, Web etc), and proxying non-HTTP 
connections, as bonuses.

A less revolutionary approach might be to add a HTTPSConnection.tunnel() 
method, that always returns the proxy’s response, but only does the SSL 
wrapping for a successful response.

--
nosy: +vadmium
versions: +Python 3.4, Python 3.5, Python 3.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Mark Lawrence]

Mark Lawrence added the comment:

Is there any work still needed here?  Surely the 2.6.x patches can't be applied 
unless there are security issues?

--
nosy: +BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Vackar Afzal]

Vackar Afzal added the comment:

I've found that for the Python2.6.x patch to play nicely with the popular 
rquests library, I've had to set some defaults on the modified __init__ 
function so that it reads as follows:


def __init__(self, *args, **kwargs):
_orig_init(self, *args, **kwargs)
self._tunnel_headers = {}
self._tunnel_host = ''
self._tunnel_port = ''


Also seems to work with python 2.6.1. Note: Change the entry condition to:

if os.environ.get('https_proxy', None) and sys.version_info[:2]  == (2, 6) :

--
nosy: +vzafzal

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Vackar Afzal]

Vackar Afzal added the comment:

Also needed to make another minor update to the monkey patch.
Have uploaded the new files as new_http_proxy_patch.py for use with python 2.6.x

--
Added file: http://bugs.python.org/file34174/new_http_proxy_patch.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Barry Scott]

Barry Scott barry.sc...@onelan.co.uk added the comment:

The attached patch builds on Mike's work.

The core of the problem is that the Request object
did not know what was going on. This means that it
was not possible for get_authorization() to work
for proxy-auth and www-auth.

I change Request to know which of the four types of
connection it represents. There are new methods on
Request that return the right information based on
the connection type.

To understand how to make this work I needed to
instrument the code. There is now a set_debuglevel
on the OpenerDirector object that turns on debug in
all the handlers and the director. I have added
more debug messages to help understand this code.

This code now passes the 72 test cases I run. I'll
attach the code I used to test as a follow up to this.

--
nosy: +b.a.scott
Added file: http://bugs.python.org/file20821/http_proxy_https.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Barry Scott]

Barry Scott barry.sc...@onelan.co.uk added the comment:

Attached is the code I used to test these changes.
See the README.txt file for details include
the results of a test run.

--
Added file: http://bugs.python.org/file20822/urllib2_tests.tar.gz

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Barry Scott]

Barry Scott barry.sc...@onelan.co.uk added the comment:

I left out some white space changes to match the style
of the std lib code. Re posting with white space cleanup.

--
Added file: http://bugs.python.org/file20824/http_proxy_https.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Barry Scott]

Changes by Barry Scott barry.sc...@onelan.co.uk:


Removed file: http://bugs.python.org/file20821/http_proxy_https.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Mike Beachy]

Mike Beachy mbea...@gmail.com added the comment:

I've been in contact w/ Barry Scott offline re: the monkey patch previously 
mentioned. I'm attaching a 2.7 maintenance branch patch that he has needed to 
extend, and plans to follow up on.

--
Added file: http://bugs.python.org/file20798/2_7_x.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Mike Beachy]

Mike Beachy mbea...@gmail.com added the comment:

Attached to this comment (can you attach multiple files at once?) is the 
somewhat moldy 2.6.4 monkey patch, mercilessly ripped from my own code and 
probably not good for much.

--
Added file: http://bugs.python.org/file20799/monkey_2_6_4.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Mike Beachy]

Mike Beachy mbea...@gmail.com added the comment:

I have worked up a monkey patch for urllib2/httplib for the issue of setting 
the authentication using a Proxy(Basic|Digest)AuthHandler.

The basic approach was to create a new httplib exception (ProxyTunnelError) and 
raise that with the http response attached so that the HTTPSHandler can 
determine when 407 Proxy authentication required is present, and then reroute 
the urllib2.OpenerDirector to error handling mode.

Unfortunately, there is a backwards compatibility issue - cases where a 
socket.error was previously being raised now get an ProxyTunnelError. Not that 
you could do much useful with the socket.error in the first place, but I 
suppose you could look for '407' in the text. Ugh.

If you think this might prove useful, let me know and I can rework it into a 
real patch - just let me know what branch/version to base it off of. (My monkey 
patch is for python 2.6.4.)

--
nosy: +mbeachy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

In this ticket, setting the authentication using a ProxyBasicAuthHandler is not 
yet addressed yet. (this was informed in the last note). Reopening this one to 
track it.

--
resolution: fixed - accepted
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Manuel Muradás]

Manuel Muradás mmura...@dieresys.com.ar added the comment:

Hi! 2.6 backport is missing an argument in _set_tunnel definition. It
should be:

def _set_tunnel(self, host, port=None, headers=None):

--
nosy: +dieresys

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Manuel Muradás]

Manuel Muradás mmura...@dieresys.com.ar added the comment:

The patch fixes only when you pass the authentication info in the proxy
handler's URL. Like:

proxy_handler = urllib2.ProxyHandler({'https':
'http://user:p...@proxy-example.com:3128/'})

But setting the authentication using a ProxyBasicAuthHandler is still
broken:

proxy_auth_handler = urllib2.ProxyBasicAuthHandler()
proxy_auth_handler.add_password('realm', 'proxy-example.com:3128',
'user', 'pass')

In the attached file (urllib2_with_proxy_auth_comparison.py) we've wrote
a comparison between what works with HTTP and HTTPS.

The problem is the 407 error never reaches the ProxyBasicAuthHandler
because HTTPConnection._tunnel raises an exception when the http
response status code is not 200.

--
Added file: 
http://bugs.python.org/file15669/urllib2_with_proxy_auth_comparison.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Thanks for the note, Manuel. Fixed it in revision 77013.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Fixed and Committed revision 76908 in the trunk.

--
keywords:  -needs review
resolution: accepted - fixed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Fixed through reversions r76908, r76909, r76910, r76911

Thanks for the patch, Tatsuhiro Tsujikawa.

--
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Ronald Oussoren]

Ronald Oussoren ronaldousso...@mac.com added the comment:

The patch looks good to me.

IMHO this should be backported to 2.6 as well.

--
keywords: +needs review
nosy: +ronaldoussoren

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Ronald Oussoren]

Ronald Oussoren ronaldousso...@mac.com added the comment:

I've tested a backport of the patch to 2.6 (just replace set_proxy by 
_set_proxy in the patch) and the resulting version of urllib2 can login to 
the proxy (as expected).

Thanks for the patch.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Senthil Kumaran]

Changes by Senthil Kumaran orsent...@gmail.com:


--
assignee:  - orsenthil
nosy: +orsenthil
resolution:  - accepted

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Tatsuhiro Tsujikawa]

Changes by Tatsuhiro Tsujikawa tsujik...@gp.hitachi.co.jp:


--
versions: +Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Tatsuhiro Tsujikawa]

New submission from Tatsuhiro Tsujikawa tsujik...@gp.hitachi.co.jp:

urllib2 cannot handle https with proxy requiring authorization.

After https_proxy is set correctly,

Python 2.6.4 (r264:75706, Oct 29 2009, 15:38:25)
[GCC 4.4.1] on linux2
Type help, copyright, credits or license for more information.
 import urllib2
 c=urllib2.urlopen(https://sourceforge.net;)
Traceback (most recent call last):
  File stdin, line 1, in module
  File /usr/lib/python2.6/urllib2.py, line 124, in urlopen
return _opener.open(url, data, timeout)
  File /usr/lib/python2.6/urllib2.py, line 389, in open
response = self._open(req, data)
  File /usr/lib/python2.6/urllib2.py, line 407, in _open
'_open', req)
  File /usr/lib/python2.6/urllib2.py, line 367, in _call_chain
result = func(*args)
  File /usr/lib/python2.6/urllib2.py, line 1154, in https_open
return self.do_open(httplib.HTTPSConnection, req)
  File /usr/lib/python2.6/urllib2.py, line 1121, in do_open
raise URLError(err)
urllib2.URLError: urlopen error Tunnel connection failed: 407 Proxy
Authentication Required

This is because HTTPConnection::_tunnel() in httplib.py doesn't send
Proxy-Authorization header.

--
components: Library (Lib)
messages: 95058
nosy: tsujikawa
severity: normal
status: open
title: urllib2 cannot handle https with proxy requiring auth
versions: Python 2.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7291] urllib2 cannot handle https with proxy requiring auth

[Tatsuhiro Tsujikawa]

Tatsuhiro Tsujikawa tsujik...@gp.hitachi.co.jp added the comment:

I created a patch.
I added additional argument 'headers' to HTTPConnection::set_tunnel()
method,
which is a mapping of HTTP headers to sent with CONNECT method. Since
authorization
credential is already set to Request object, in
AbstractHTTPHandler::do_open(),
if Proxy-Authorization header is found, pass it to set_tunnel().

It works fine for me.

--
keywords: +patch
Added file: http://bugs.python.org/file15296/https_proxy_auth.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7291
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com