Hello everyone, I'm trying to follow redirections using scrapy, however i'm having the following issue:
if i don't set an user-agent for the t.co url shortener, i get 301 status code and this response body: # 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None, <noscript><META http-equiv="refresh" content="0;URL=http://sn.im/29m4uw5";></ noscript><title>http://sn.im/29m4uw5</title><script>window.opener = null; location.replace("http:\/\/sn.im\/29m4uw5")</script> now, when i set an user agent (*PS 3*), i get 200 status code and same response body (and redirection isn't solved). I need to set an user-agent to track chain redirections, like if i visit them. So, i need solve redirections although the HTTP status code don't be 3XX (like in this case, 200). Thank you for your help! *PS*: the url used for testing is http://t.co/b00VzIsr6l *PS 2*: would i need something like scrapyjs <https://github.com/scrapinghub/scrapyjs>? *PS 3*: scrapy project settings are: DOWNLOADER_MIDDLEWARES = { 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': 400, 'scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware': 550, 'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': 600, } REDIRECT_ENABLED = True METAREFRESH_ENABLED = True USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36' -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
