I usually monitor the browser reporting to see what bots might be slipping through, but I haven't checked for a while and in looking you may like to add "spider" to that list. There are also a lot of hits from
"Mozilla/2.0 (compatible; Ask Jeeves/Teoma;
+http://sp.ask.com/docs/about/tech_crawling.html)"
so maybe change "crawler" to simply "crawl".
The short answer is that it is good enough for what I need. And in looking at
the browser and bot reports it is catching at least 98%.
B)
Andrew Scott wrote:
Brett,
How effective is this method, do you just keep adding or will it catch 98%
of bots?
Regards
Andrew Scott
Analyst Programmer
CMS Transport Systems
Level 2/33 Bank Street
South Melbourne, Victoria, 3205
Phone: 03 9699 7988 - Fax: 03 9699 7976
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Brett
Payne-Rhodes
Sent: Wednesday, 13 July 2005 12:38 PM
To: CFAussie Mailing List
Subject: [cfaussie] RE: Search Engines
This is what I use to try and identify 'bots'
<!--- Check browser for 'bot' identifiers... --->
<cfset botList =
"bot,crawler,ia_archiver,pompos,WebRescuer,/ysearch/,ZyBorg">
<cfloop list="#botList#" index="botName">
<cfif (findNoCase(botName, cgi.http_user_agent) gt 0)>
<cfset session.botSession = true>
<cfbreak>
</cfif>
</cfloop>
No warranty or guarantee implied... ;)
B)
Andrew Scott wrote:
Guys,
Please I am not interested in the ins and outs of search engines. I really
need this list for the project and that's it.
I have been trying to solve this problem for now 6 months, and the
robots.txt file is correct, and I need to stop them from accessing content
in my code and my only option due to the way this website has been written
is to see if the search engine has arrived and stop them from displaying
certain content in a page.
Regards
Andrew Scott
Analyst Programmer
CMS Transport Systems
Level 2/33 Bank Street
South Melbourne, Victoria, 3205
Phone: 03 9699 7988 - Fax: 03 9699 7976
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Dawes,
Chris
Sent: Wednesday, 13 July 2005 12:16 PM
To: CFAussie Mailing List
Subject: [cfaussie] RE: Search Engines
Have you set the file up correctly?
http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Andrew
Scott
Sent: Wednesday, 13 July 2005 12:16 PM
To: CFAussie Mailing List
Subject: [cfaussie] RE: Search Engines
I have that as well, but they are ignoring this file.
Regards
Andrew Scott
Analyst Programmer
CMS Transport Systems
Level 2/33 Bank Street
South Melbourne, Victoria, 3205
Phone: 03 9699 7988 - Fax: 03 9699 7976
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Dawes,
Chris
Sent: Wednesday, 13 July 2005 12:07 PM
To: CFAussie Mailing List
Subject: [cfaussie] RE: Search Engines
Once again robots.txt is your friend... It tells the spiders where to
look and where NOT to look...
Dawesi
---
You are currently subscribed to cfaussie as: [EMAIL PROTECTED]
To unsubscribe send a blank email to
[EMAIL PROTECTED]
Aussie Macromedia Developers: http://lists.daemon.com.au/
----------------------------------------------------
This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
Please note that any views or opinions presented in this email are solely
those of the author and do not necessarily represent those of the company.
The recipient should check this email and any attachments for the presence
of viruses. The company accepts no liability for any damage caused by any
virus transmitted by this email.
----------------------------------------------------
---
You are currently subscribed to cfaussie as: [EMAIL PROTECTED]
To unsubscribe send a blank email to
[EMAIL PROTECTED]
Aussie Macromedia Developers: http://lists.daemon.com.au/
---
You are currently subscribed to cfaussie as: [EMAIL PROTECTED]
To unsubscribe send a blank email to
[EMAIL PROTECTED]
Aussie Macromedia Developers: http://lists.daemon.com.au/
---
You are currently subscribed to cfaussie as: [EMAIL PROTECTED]
To unsubscribe send a blank email to
[EMAIL PROTECTED]
Aussie Macromedia Developers: http://lists.daemon.com.au/
---
You are currently subscribed to cfaussie as: [EMAIL PROTECTED]
To unsubscribe send a blank email to [EMAIL PROTECTED]
Aussie Macromedia Developers: http://lists.daemon.com.au/
---
You are currently subscribed to cfaussie as: [email protected]
To unsubscribe send a blank email to [EMAIL PROTECTED]
Aussie Macromedia Developers: http://lists.daemon.com.au/
