Hey everyone, having issues scraping a site which keeps on giving me a 429 response. After checking the site in scrapy shell and looking at the response I noticed that even if I fetch only the first page in scrapy shell, I'm presented with a security question a la "oops this looks suspicious prove that your are not a robot". I have changed the User-Agent, I'm using ProxyMesh, have disabled cookies and tried multiple download delays. Now I'm out of ideas. In my browser everything is as it should be.
Something de-masks my bot as a bot. Any ideas of how to tackle this? -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
