Hi my friends I want to process redirect just to extract the "to URL", I
tried :
meta = {'dont_redirect': True,'handle_httpstatus_list': [301]}
But it doesn't work, I tested with [404, 301], the result is that just 404
work.
http://stackoverflow.com/questions/15476587/scrapy-how-to-stop-redirect-302
* The content of redirect_urls in meta doesn't work for me, because I
should have to crawl all the redirected pages until to know that is has a
redirect. That caused me a performance problem crawling even already
crawled pages, just to know if it comes from a redirect.
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to scrapy-users+unsubscr...@googlegroups.com.
To post to this group, send email to scrapy-users@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
