On Thursday 17 Jul 2003 17:49 pm, Jim McAtee wrote: > page I'm logging on web site "B", and web site B has a robot.txt file > telling the bots to not index the page on B, will the search engine still > follow the link and request the page on B? What I'm interested in isn't so
A well written bot should examine /robots.txt for all hosts, before requesting any files from it. -- Thomas C Advanced ColdFusion Programmer HANDLE WITH EXTREME CARE: This Email Contains Minute Electrically Charged Particles Moving at Velocities in Excess of Five Hundred Million Miles Per Hour. PLEASE ALSO NOTE: I don't speak for the company that sent this. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Archives: http://www.houseoffusion.com/cf_lists/index.cfm?forumid=4 Subscription: http://www.houseoffusion.com/cf_lists/index.cfm?method=subscribe&forumid=4 FAQ: http://www.thenetprofits.co.uk/coldfusion/faq This list and all House of Fusion resources hosted by CFHosting.com. The place for dependable ColdFusion Hosting. http://www.cfhosting.com Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
