Dealing With Spiders/Bots/Crawlers

2010-02-03 Thread Bob Hendren
I'm at an odd crossroads here. Up until now, I've kept my application off-limits to search engines. I've used a couple of techniques found on Ben Nadel's blog for giving them short sessions and such. Been working well. With respect to human users, I've been VERY diligent about using

Dealing with Spiders/Bots/Crawlers

2010-02-03 Thread Bob Hendren
I'm at an odd crossroads here. Up until now, I've kept my application off-limits to search engines. I've used a couple of techniques found on Ben Nadel's blog for giving them short sessions and such. Been working well. With respect to human users, I've been VERY diligent about using

RE: Dealing With Spiders/Bots/Crawlers

2010-02-03 Thread brad
info in all your URLs) There are other pitfalls with placing the session IDs in every URL other than bot traffic: http://kb2.adobe.com/cps/172/tn_17255.html ~Brad Original Message Subject: Dealing With Spiders/Bots/Crawlers From: Bob Hendren bhend...@listingware.com Date: Wed

Re: Dealing With Spiders/Bots/Crawlers

2010-02-03 Thread Bob Hendren
First, sorry for dual post earlier. Second, not a stupid question about cookies, but just trying to deal with the possibility that a user has cookies turned off. I'm using URLSessionFormat on all of my links to pass their session info around so the system isn't constantly treating them like

Re: Dealing with Spiders/Bots/Crawlers

2010-02-03 Thread Mike Chabot
Be careful about setting all tokens to 1 if your Web site contains sensitive information and if you are putting this info on a URL string. What will happen is that every user that comes through via a search engine will be considered the same user according to ColdFusion, which is rather bad. If

RE: Dealing With Spiders/Bots/Crawlers

2010-02-03 Thread brad
just trying to deal with the possibility that a user has cookies turned off. I see. Well I've pretty much said my opinion on the cookie thing, but to reiterate, people not using cookies are such a minute percentage of the web. Unfortunately, I can't find any recent numbers anywhere, but I'm