Good points. Perhaps we should integrate your function with fetch. On Feb 1, 2:38 am, Richard <[email protected]> wrote: > If I understand correctly, fetch() was added to web2py because at the > time GAE did not support urllib2. > Now GAE does support urllib2 which makes fetch() sort of redundant, > but we need to keep it for backward compatibility. > > I found that urllib2 on GAE does not handle cookies as normal, so have > been using the function below. Would it be useful to replace fetch() > with something like this? > > import urllib > import urllib2 > urllib2.install_opener(urllib2.build_opener(urllib2.HTTPCookieProcessor > ())) > import Cookie > cookie = Cookie.SimpleCookie() > > def download(url, data=None, user_agent='Mozilla/5.0'): > data = data if data is None else urllib.urlencode(data) > headers = {'User-agent': user_agent} > try: > from google.appengine.api import urlfetch > except ImportError: > request = urllib2.Request(url, data, headers) > html = urllib2.urlopen(request).read() > else: > headers['Cookie'] = ' '.join('%s=%s;' % (c.key, c.value) for c > in cookie.values()) > method = urlfetch.GET if data is None else urlfetch.POST > while url is not None: > response = urlfetch.fetch(url=url, payload=data, > method=method, headers=headers, allow_truncated=False, > follow_redirects=False, deadline=10) > data = None # next request will be a get, so no need to > send the data again > method = urlfetch.GET > cookie.load(response.headers.get('set-cookie', '')) # load > cookies from the response > url = response.headers.get('location') > html = response.content > return html
-- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
