On Monday 23 June 2003 10:03 am, Jared Odulio wrote:
> what if the bot is using an anonymous access or mimicking another a
> legit bot name? Bot exclusion files are just one of the most subtle way
> of thwarting bot access.
legitimate search engines will honor robots.txt. however,
not all crawlers are legitimate. or the unauthorized user
might just be using wget (modified to ignore robots.txt)
to download your entire site.
is password protecting the site an option?
tiger
--
Gerald Timothy Quimpo gquimpo*hotmail.com tiger*sni*ph
Public Key: "gpg --keyserver pgp.mit.edu --recv-keys 672F4C78"
Premature optimization is the root of all evil.
Donald Knuth
--
Philippine Linux Users' Group (PLUG) Mailing List
[EMAIL PROTECTED] (#PLUG @ irc.free.net.ph)
Official Website: http://plug.linux.org.ph
Searchable Archives: http://marc.free.net.ph
.
To leave, go to http://lists.q-linux.com/mailman/listinfo/plug
.
Are you a Linux newbie? To join the newbie list, go to
http://lists.q-linux.com/mailman/listinfo/ph-linux-newbie
