Maybe rename the file "donotdownloadandinstallmedumbass.tar.gz" just to be
clear? :)
robots.txt is a file that goes in your docroot and contains a listing of
directories and files that you don't want google and other search engines
to spider/index. Usually you add entries such as your cgi-bin dir and so
forth. We are using it for a client site that has different sales each
day. No point in indexing those since they change, might as well not
waste googles time on it. There isn't a requirement for it but having it
there is part of being a 'responsible webmaster'.
On Tue, 16 Apr 2002, John Hunter wrote:
> >>>>> "tack" == tack <[EMAIL PROTECTED]> writes:
>
> That said, where should ROBOTS.TXT be placed, and what should it
> contain?
>
> JDH
-e
_____________________________________________________________________________
<majcher> icky is like a shadowy hit man, that nobody ever sees, and can only
contact through some strange process.
_______________________________________________
Bits mailing list
[EMAIL PROTECTED]
http://www.sugoi.org/mailman/listinfo/bits
