thanks for correction.
On Sun, May 15, 2011 at 10:57 PM, Stephen <[email protected]> wrote: > On Sat, May 14, 2011 at 6:12 AM, Gopal Patel <[email protected]> wrote: > > there is exabot crawler crawling my unfinished website > continuously....what > > can I do to stop it ? I have disallowed everything in robot.txt . > > That should be robots.txt, with an 's'. > > They claim it respects robots.txt: > > > http://www.exalead.com/software/services/knowledgebase/websearchengine/webmasterguide/#how-may-i-exclude-my-site-from-being-crawled-by-the-exalead-robot > > But it seems to be a badly behaved bot in general, so you may want to > add it's IP addresses to your dos.yaml file: > > http://code.google.com/appengine/docs/python/config/dos.html > > (This only blocks access to dynamic pages) > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
