Permissions! ====================================== Stop spam on your domain, use our gateway! For hosting solutions http://www.clickdoug.com ISP rated: http://www.forta.com/cf/isp/isp.cfm?isp_id=772 ====================================== If you are not satisfied with my service, my job isn't done!
----- Original Message ----- From: "Randell B Adkins" <[EMAIL PROTECTED]> To: "CF-Community" <[EMAIL PROTECTED]> Sent: Thursday, July 17, 2003 11:06 AM Subject: Re: Search Engine / robot.txt | Then what is the best way to prohibit directories from | being searched by those bots? | | So far I have wrote the robots.txt file to allow all bots | and DISALLOW all directories and files with the | exception of the index.cfm file. | | Just a lot of DISALLOW commands | | | | >>> [EMAIL PROTECTED] 07/17/03 11:53AM >>> | Just remember, that many of the search bots totally ignore the | Robots.txt files | as well as the meta tags. | | inktomi is one of the major offenders. | | ====================================== | Stop spam on your domain, use our gateway! | For hosting solutions http://www.clickdoug.com | ISP rated: http://www.forta.com/cf/isp/isp.cfm?isp_id=772 | ====================================== | If you are not satisfied with my service, my job isn't done! | | ----- Original Message ----- | From: "Critz" <[EMAIL PROTECTED]> | To: "CF-Community" <[EMAIL PROTECTED]> | Sent: Thursday, July 17, 2003 8:11 AM | Subject: Re: Search Engine / robot.txt | | | | oi Randell!! | | | | yeah, I don't even know if they will index the url.... but they go | no further | than that....(if they | | use the robots.txt file....) | | | | | | | | | | ------------------------------------ | | Thursday, July 17, 2003, 8:57:55 AM, you wrote: | | | | RBA> Thanks also by using: | | RBA> User-agent: * | | RBA> Disallow: / | | | | RBA> That would allow all BOTs to hit your site but not index any | pages. | | RBA> Is that right? so in theory it would index just the URL and the | | RBA> META tag data? | | | | | | | | | | | | >>>> [EMAIL PROTECTED] 07/17/03 08:49AM >>> | | RBA> oi Randell!! | | | | RBA> | http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=how+to+build+a+robots.txt&btnG=Google+Search | | | | | | | RBA> first return looks like it might splain it... | | | | RBA> mine all just say: | | | | RBA> User-agent: * | | RBA> Disallow: / | | | | | | RBA> :) | | | | | | | | | | RBA> ------------------------------------ | | RBA> Thursday, July 17, 2003, 8:43:30 AM, you wrote: | | | | RBA>> Does anyone have a good resource for building a robot.txt | | RBA>> file for search engine bots? | | | | | | RBA>> | | | | RBA> | | | | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Archives: http://www.houseoffusion.com/cf_lists/index.cfm?forumid=5 Subscription: http://www.houseoffusion.com/cf_lists/index.cfm?method=subscribe&forumid=5 Signup for the Fusion Authority news alert and keep up with the latest news in ColdFusion and related topics. http://www.fusionauthority.com/signup.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.5
