The best way is to remove it from your directory from the google webmaster
tools. Also some bots don't listen so additionally to robots.txt use the
webmaster central.
James
On Sun, Jan 17, 2010 at 9:31 AM, Kai Schaetzl mailli...@conactive.comwrote:
Add
User-agent: Slurp
Crawl-delay: 86400
On Sat, Jan 16, 2010 at 10:18 PM, R-Elists list...@abbacomm.net wrote:
would anyone out there care to share their robots.txt experience using
centos as a webserver and their robots.txt files?
i realize this is a somewhat simple exercise, yet i am sure there are both
large and small hosters
On Sat, 2010-01-16 at 14:18 -0800, R-Elists wrote:
quite frankly, there are many orgs out there that dont follow this
anyways,right?
Since robots.txt is a suggestion and .htaccess is actually enforced, I
use a simple robots.txt like this:
User-agent: *
Disallow:
and put the bad guys into
3 matches
Mail list logo
