Nick is right. If you want to have an API, you shouldn’t block based on the method of “curl” being used, but by something more like user/pass, OAUTH, Token, or any of a dozen other methods.
If you are just talking about a Web site not an API, then I could see preventing scraping by saying “nothing but Humans and the big 3 search bots” I have done this to sites in the past. Of course you then also have to check for “browsing faster than humans do. Or “not rendering the page” I had a site that we wanted users to sign up for coupons ala groupon and we had trouble with other people snarfing the deals of the day and the promo codes so we implemented a “you have to download this .js file contained in the HTML once every 3 page views” worked well, even if it was secure through obscure. -Brandon From: [email protected] [mailto:[email protected]] On Behalf Of Álvaro Degives-Más Sent: Wednesday, April 06, 2011 1:03 AM To: [email protected] Subject: Re: [google-appengine] How to block Google App Engine? Hi Nick, that's excellent advice, and a good analysis from the quantitative end of things. The problem is that abuse does not manifest itself in a statistically more or less uniform manner, so as to allow applying risk factoring as you suggest it. It's an issue that goes far beyond Google App Engine; I was merely looking for ways to strike a fair balance. But in the end, cost-benefit advises against the greater granularity you also suggest. To put it in another analogy: sometimes, disconnecting the line is the more efficacious solution for harassment calls. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
