And not include what it finds in its index? Adrian
-----Original Message----- From: Michael Dinowitz Sent: 12 October 2008 22:51 To: cf-talk Subject: Re: How does Security affect search engine spiders? There is an option in adsense to bypass login based security in order to index pages for ads. While your pages may not have ads on them, using this option guarantees that Google will get through your security. On 10/12/08, Doug Boude (rhymes with 'loud') <[EMAIL PROTECTED]> wrote: > > Hi all. I am curious if anybody knows how securing a site affects a search > engine spider's ability to crawl it. For instance, if I have my entire site > secured by means of authentication so that any page request is redirected to > the login page if the appropriate security creds are not present in session, > do spiders receive the same treatment? Are they also prohibited by my > security from crawling any page except the login page? If this is true, what > can I do to allow spiders to have access to crawl content but still apply > security to regular "human" visitors? My only thought on that is to detect > the fact that they are a spider (not sure how to do that though) and not > implement security in that case. > > Thanks for your ideas and thoughts. Feel free to email them to me at > [EMAIL PROTECTED] > > Doug :0) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:313793 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
