I know there is mail-abuse for listing open relay sites.
Is there an equivalent for bad search engine bots?
I currently have three search engines crawling my www and all three
ignore robots.txt. This means that they are determined to index about
40,000 stored Slug, 2600, htdig, etc messages. Two of them are working
at a fair clip as well.
A little Apache add-on that delayed the reply to requests from
particular IPs would be most welcome.
--
Terry Collins {:-)}}} Ph(02) 4627 2186 Fax(02) 4628 7861
email: [EMAIL PROTECTED] www: http://www.woa.com.au
WOA Computer Services <lan/wan, linux/unix, novell>
"People without trees are like fish without clean water"
--
SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/
More Info: http://slug.org.au/lists/listinfo/slug
