No that's good because it was what I was thinking as well, and seems like it will match my needs as well.
Regards Andrew Scott Analyst Programmer CMS Transport Systems Level 2/33 Bank Street South Melbourne, Victoria, 3205 Phone: 03 9699 7988 - Fax: 03 9699 7976 -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Payne-Rhodes Sent: Wednesday, 13 July 2005 1:16 PM To: CFAussie Mailing List Subject: [cfaussie] RE: Search Engines I usually monitor the browser reporting to see what bots might be slipping through, but I haven't checked for a while and in looking you may like to add "spider" to that list. There are also a lot of hits from "Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)" so maybe change "crawler" to simply "crawl". The short answer is that it is good enough for what I need. And in looking at the browser and bot reports it is catching at least 98%. B) Andrew Scott wrote: > Brett, > > How effective is this method, do you just keep adding or will it catch 98% > of bots? > > > Regards > Andrew Scott > Analyst Programmer > > CMS Transport Systems > Level 2/33 Bank Street > South Melbourne, Victoria, 3205 > > Phone: 03 9699 7988 - Fax: 03 9699 7976 > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett > Payne-Rhodes > Sent: Wednesday, 13 July 2005 12:38 PM > To: CFAussie Mailing List > Subject: [cfaussie] RE: Search Engines > > > This is what I use to try and identify 'bots' > > <!--- Check browser for 'bot' identifiers... ---> > <cfset botList = > "bot,crawler,ia_archiver,pompos,WebRescuer,/ysearch/,ZyBorg"> > <cfloop list="#botList#" index="botName"> > <cfif (findNoCase(botName, cgi.http_user_agent) gt 0)> > <cfset session.botSession = true> > <cfbreak> > </cfif> > </cfloop> > > No warranty or guarantee implied... ;) > > B) > > > Andrew Scott wrote: > >>Guys, >> >>Please I am not interested in the ins and outs of search engines. I really >>need this list for the project and that's it. >> >>I have been trying to solve this problem for now 6 months, and the >>robots.txt file is correct, and I need to stop them from accessing content >>in my code and my only option due to the way this website has been written >>is to see if the search engine has arrived and stop them from displaying >>certain content in a page. >> >> >>Regards >>Andrew Scott >>Analyst Programmer >> >>CMS Transport Systems >>Level 2/33 Bank Street >>South Melbourne, Victoria, 3205 >> >>Phone: 03 9699 7988 - Fax: 03 9699 7976 >> >>-----Original Message----- >>From: [EMAIL PROTECTED] >>[mailto:[EMAIL PROTECTED] On Behalf Of Dawes, > > Chris > >>Sent: Wednesday, 13 July 2005 12:16 PM >>To: CFAussie Mailing List >>Subject: [cfaussie] RE: Search Engines >> >> >>Have you set the file up correctly? >> >>http://www.searchengineworld.com/cgi-bin/robotcheck.cgi >> >>-----Original Message----- >>From: [EMAIL PROTECTED] >>[mailto:[EMAIL PROTECTED] On Behalf Of Andrew >>Scott >>Sent: Wednesday, 13 July 2005 12:16 PM >>To: CFAussie Mailing List >>Subject: [cfaussie] RE: Search Engines >> >>I have that as well, but they are ignoring this file. >> >>Regards >>Andrew Scott >>Analyst Programmer >> >>CMS Transport Systems >>Level 2/33 Bank Street >>South Melbourne, Victoria, 3205 >> >>Phone: 03 9699 7988 - Fax: 03 9699 7976 >> >>-----Original Message----- >>From: [EMAIL PROTECTED] >>[mailto:[EMAIL PROTECTED] On Behalf Of Dawes, >>Chris >>Sent: Wednesday, 13 July 2005 12:07 PM >>To: CFAussie Mailing List >>Subject: [cfaussie] RE: Search Engines >> >> >>Once again robots.txt is your friend... It tells the spiders where to >>look and where NOT to look... >> >>Dawesi >> >> >> >> >>--- >>You are currently subscribed to cfaussie as: [EMAIL PROTECTED] >>To unsubscribe send a blank email to >>[EMAIL PROTECTED] >>Aussie Macromedia Developers: http://lists.daemon.com.au/ >> >>---------------------------------------------------- >>This email and any files transmitted with it are confidential and intended >>solely for the use of the individual or entity to whom they are addressed. >>If you have received this email in error please notify the system manager. >>Please note that any views or opinions presented in this email are solely >>those of the author and do not necessarily represent those of the company. >>The recipient should check this email and any attachments for the presence >>of viruses. The company accepts no liability for any damage caused by any >>virus transmitted by this email. >>---------------------------------------------------- >> >>--- >>You are currently subscribed to cfaussie as: [EMAIL PROTECTED] >>To unsubscribe send a blank email to >>[EMAIL PROTECTED] >>Aussie Macromedia Developers: http://lists.daemon.com.au/ >> >> >> >> >>--- >>You are currently subscribed to cfaussie as: [EMAIL PROTECTED] >>To unsubscribe send a blank email to > > [EMAIL PROTECTED] > >>Aussie Macromedia Developers: http://lists.daemon.com.au/ >> > > > --- > You are currently subscribed to cfaussie as: [EMAIL PROTECTED] > To unsubscribe send a blank email to > [EMAIL PROTECTED] > Aussie Macromedia Developers: http://lists.daemon.com.au/ > > > > > --- > You are currently subscribed to cfaussie as: [EMAIL PROTECTED] > To unsubscribe send a blank email to [EMAIL PROTECTED] > Aussie Macromedia Developers: http://lists.daemon.com.au/ > --- You are currently subscribed to cfaussie as: [EMAIL PROTECTED] To unsubscribe send a blank email to [EMAIL PROTECTED] Aussie Macromedia Developers: http://lists.daemon.com.au/ --- You are currently subscribed to cfaussie as: [email protected] To unsubscribe send a blank email to [EMAIL PROTECTED] Aussie Macromedia Developers: http://lists.daemon.com.au/
