Then what is the best way to prohibit directories from being searched by those bots?
So far I have wrote the robots.txt file to allow all bots and DISALLOW all directories and files with the exception of the index.cfm file. Just a lot of DISALLOW commands >>> [EMAIL PROTECTED] 07/17/03 11:53AM >>> Just remember, that many of the search bots totally ignore the Robots.txt files as well as the meta tags. inktomi is one of the major offenders. ====================================== Stop spam on your domain, use our gateway! For hosting solutions http://www.clickdoug.com ISP rated: http://www.forta.com/cf/isp/isp.cfm?isp_id=772 ====================================== If you are not satisfied with my service, my job isn't done! ----- Original Message ----- From: "Critz" <[EMAIL PROTECTED]> To: "CF-Community" <[EMAIL PROTECTED]> Sent: Thursday, July 17, 2003 8:11 AM Subject: Re: Search Engine / robot.txt | oi Randell!! | | yeah, I don't even know if they will index the url.... but they go no further than that....(if they | use the robots.txt file....) | | | | | ------------------------------------ | Thursday, July 17, 2003, 8:57:55 AM, you wrote: | | RBA> Thanks also by using: | RBA> User-agent: * | RBA> Disallow: / | | RBA> That would allow all BOTs to hit your site but not index any pages. | RBA> Is that right? so in theory it would index just the URL and the | RBA> META tag data? | | | | | | >>>> [EMAIL PROTECTED] 07/17/03 08:49AM >>> | RBA> oi Randell!! | | RBA> http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=how+to+build+a+robots.txt&btnG=Google+Search | | | RBA> first return looks like it might splain it... | | RBA> mine all just say: | | RBA> User-agent: * | RBA> Disallow: / | | | RBA> :) | | | | | RBA> ------------------------------------ | RBA> Thursday, July 17, 2003, 8:43:30 AM, you wrote: | | RBA>> Does anyone have a good resource for building a robot.txt | RBA>> file for search engine bots? | | | RBA>> | | RBA> | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Archives: http://www.houseoffusion.com/cf_lists/index.cfm?forumid=5 Subscription: http://www.houseoffusion.com/cf_lists/index.cfm?method=subscribe&forumid=5 Get the mailserver that powers this list at http://www.coolfusion.com Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.5
