You use an off Google re-proxy. Which will also help when the site you are scraping sees that it is getting scraped and blocks the Google Fetches User Agent.
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of csvy Sent: Sunday, September 27, 2009 2:46 PM To: Google App Engine Subject: [google-appengine] URLFetch Proxy Discarding Most of Set-Cookie? I'm trying to use URLFetch in Python to scrape a website that has a login. The code works great locally in the GAE dev environment, however it doesn't work when I upload it to the cloud I'm figured out exactly what the issue is, but I have no idea how to prevent it from happening. When I make a URLFetch request on my local machine, the Set-Cookie header includes 6 cookies. When I make the same request on the server, the Set-Cookie header only includes 1 cookie (the last of the 6). Why is the Google proxy removing the cookies and what do I do about it??? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
