On 2008-03-02 21:11, Brett Cannon wrote:
> Well, look at the docs for urllib. There is a list of restrictions
> (e.g., does not support the use of proxies which require
> authentication). From what I can tell, those items on the list that
> are an actual restriction do not carry over to urllib2.
I'm not sure I follow you: urllib *does* support proxies that
require authentication (see the .open_http() method).
According to http://docs.python.org/dev/library/urllib.html#module-urllib:
"This module does not support the use of proxies which require authentication".
Looks like the documentation is not up-to-date:
...
if proxy_auth: h.putheader('Proxy-Authorization', 'Basic %s' %
proxy_auth)
...
> Another thing,
> how do you add a custom line to the header for the request in urllib
> (e.g., Referer)? The docs for URLOpener don't seem to provide a way.
> urllib2, on the other hand, has a very specific way to add headers.
That's easy:
class URLReader(urllib.URLopener):
# Crawler name
agentname = 'mxHTMLTools-Crawler'
def __init__(*args):
""" Add a user-agent header to the HTTP requests.
"""
self = args[0]
apply(urllib.URLopener.__init__, args)
# Override the default settings for self.addheaders:
assert len(self.addheaders) == 1
self.addheaders = [
('user-agent', '%s/%s' % (self.agentname, HTMLTools.__version__)),
]
...
But none of that is documented. So if the classes do stay then they
really need to have their documentation flushed out (along with making
sure they have the proper unit tests for those exposed APIs, of
course).
Agreed.
> But as I said in my last email, I am happy to include URLOpener if
> some other people are willing to back the idea up.
Fair enough.
I will send a separate email to the SIG since people have probably
stopped following most of this thread. =)
Please also CC the stdlib SIG again... that CC got lost somewhere
in the thread :-)
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Mar 03 2008)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
_______________________________________________
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
