On Sun, 9 Feb 2003, Ken Moffat wrote: > Anyone know the function of robots.txt? I have seen attempted access to > it in my apache logs. >
Yes. That is where you can limit the browsing of well-behaved robots. For instance, I run a website with a database of millions of nearly- identical dynamic pages. There's no point in letting random webcrawlers try to index them all -- it wastes their time and my bandwidth, so I put stuff in there to limit their activity. Some crawlers have their own rules, some don't obey any, but robots.txt is pretty common, and fairly standard. ++ kevin -- Kevin O'Gorman, PhD (805) 650-6274 mailto:[EMAIL PROTECTED] Permanent e-mail forwarder: mailto:Kevin.O'[EMAIL PROTECTED] Permanent e-mail forwarder mailto:[EMAIL PROTECTED] Web: http://kosmanor.com/~kevin/index.html _______________________________________________ Linux-users mailing list [EMAIL PROTECTED] Unsubscribe/Suspend/Etc -> http://www.linux-sxs.org/mailman/listinfo/linux-users
