If user agent = exabot return self error 510
(you will have to clean that up it is not code) From: [email protected] [mailto:[email protected]] On Behalf Of Gopal Patel Sent: Friday, May 13, 2011 10:13 PM To: [email protected] Subject: [google-appengine] How can I block bot which do not follow robots.txt ? there is exabot crawler crawling my unfinished website continuously....what can I do to stop it ? I have disallowed everything in robot.txt . -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
