Date: 24.07.02
Sender: Roger Williams
Time: 16:08
> Do you know of any html tag or meta-tag which will stop robots from
> accessing a web page?
> I currently use the meta-tag "no index, no follow," but I know that few
> robots actually support that.
If you want to disallow indexing of a small amount of web pages, you can
use the existing tag. I doubt on your assumption that "few robots actually
support that", since last time I checked (about a week ago) all the major
web-robots(*) did obey the rules.
(*) Google, Yahoo, inktomi, northernlight, to name a few.
> The point is to deny harvesting or indexing of certain pages by robots
> on a web site. Anybody have any solutions?
If you have administrator priviledges on the web server you can try the
"robots.txt" solution that Peter suggested. With robots.txt you can
disallow indexing of whole directories.
Another way (a little bit more efficient, yet slightly "fascist") is to
prohibit access via the "htaccess" file. You can use the "htaccess" file
in two ways:
a) By installing it in the root directory of the web server
(Administrative priviledges required)
b) By installing it in the same (or parent) directory where your "secret"
pages exist.
A sample .htaccess file could be:
-------------------------------cut----------------------------------------
AuthUserFile nonexistent
AuthGroupFile nonexistent
AuthName "Bot blocker"
AuthType Basic
ErrorDocument 403 http://www.ourserver.ourdomain/no-bots-here.html
<Limit GET>
order allow,deny
allow from all
deny from A.B.C.D/MN
deny from E.F.G.H/OP
deny from I.J.K.L/QR
</Limit>
-------------------------------cut----------------------------------------
Where A.B.C.D/MN the IP ranges that bots use while indexing the Web. This
might not be the easiest task to do but it forbids web indexing by-host.
The htaccess file is used mainly on Apache web servers. I am not aware of
what other servers use for this occasion but they have to use a similar
implementation.
IMO, you can safely use the meta tags or the robots.txt; it is sufficient.
Good luck.
--
[ Quote #217 from pccodes collection. ]
TSZ: Thus Spake Zarathustra
============= PCWorks Mailing List =================
Don't see your post? Check our posting guidelines &
make sure you've followed proper posting procedures,
http://pcworkers.com/rules.htm
Contact list owner <[EMAIL PROTECTED]>
Unsubscribing and other changes: http://pcworkers.com
=====================================================
