I'm trying to setup scrapy + tor
I'm using scrapy 0.24.6 1* I first tried using polipo to be able to access tor as an http proxy ( https://pkmishra.github.io/blog/2013/04/16/scrapy-run-using-tor-and-multiple-agents-part-2-ubuntu/ ) I'm able to configure my web browser to use polipo and I'm able to browse using TOR and I can curl. I tried HttpProxyMiddleware and using env var: scrapy returns 403 I also tried to write my own custom middleware, same result: scrapy always returns 403 2* Then I tried to use tor directly, again I can configure my web browser to use the socks proxy and I can curl with torsocks, but scrapy always returns 403 Anyone has an idea about what might be wrong ? It looks like the error come from scrapy because I have the exact same headers/user-agent with and without tor, but through tor I always get a 403 -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
