what if the bot is using an anonymous access or mimicking another a legit bot name? Bot exclusion files are just one of the most subtle way of thwarting bot access.
On Mon, 2003-06-23 at 17:21, Brian Baquiran wrote: > Aris Santillan wrote: > > > hi guys > > > > how can i avoid web crawlers > > to get info on my restricted web > > directory? > > Create a file robots.txt in your document root. The format is explained at > http://www.robotstxt.org/wc/norobots.html > > Brian > > -- > Philippine Linux Users' Group (PLUG) Mailing List > [EMAIL PROTECTED] (#PLUG @ irc.free.net.ph) > Official Website: http://plug.linux.org.ph > Searchable Archives: http://marc.free.net.ph > To leave, go to http://lists.q-linux.com/mailman/listinfo/plug > Are you a Linux newbie? To join the newbie list, go to > http://lists.q-linux.com/mailman/listinfo/ph-linux-newbie
signature.asc
Description: This is a digitally signed message part
-- Philippine Linux Users' Group (PLUG) Mailing List [EMAIL PROTECTED] (#PLUG @ irc.free.net.ph) Official Website: http://plug.linux.org.ph Searchable Archives: http://marc.free.net.ph . To leave, go to http://lists.q-linux.com/mailman/listinfo/plug . Are you a Linux newbie? To join the newbie list, go to http://lists.q-linux.com/mailman/listinfo/ph-linux-newbie
