On Wed, 16 Jan 2008 09:04:53 -0500 "Levy, Alan" <[EMAIL PROTECTED]> wrote:
> Isn't a robots.txt only for apache (which I do not use) ? The robots.txt file is just a text/plain file. The robots W3C stuff says a properly behaving robot should pull /robots.txt and honour the contexts if it is found. It asks the crawler to behave politely according to the content and most will do so. Thus it's not at all server dependant. Alan ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
