2008/10/8 <[EMAIL PROTECTED]>: > Christopher Sawtell wrote: >> >> http://www.robotstxt.org/ >> >> Google obeys, but other spiders may not. > > Thanks :) n.p.
>>> What's the best way to make apache look at the refer(?) and send it >>> packing? >> >> Reject the IP # in your firewall. > > Is there a list of all the known major engines? In the link there is:- http://www.robotstxt.org/db.html which lists approximately 128 Install a robots.txt first and then block the remaining ones in your firewall. -- Sincerely etc. Christopher Sawtell
