if i remember correctly you can put all the pages you dont want into a folder, then tell the spider not to index this folder using robots.txt.
This would prevent all robots from indexing a folder called private: User-agent: * Disallow: /private/ You can also specify pages: User-agent: * Disallow: /private/secret.html HTH will -----Original Message----- From: Craig Dudley [mailto:[EMAIL PROTECTED]] Sent: 11 December 2001 11:51 To: CF-Talk Subject: RE: OT: Keep search engine spyders from following links Doesn't robots.txt do the exact job you are after? I don't know the systnax, but any visting spider will most likely request that file 1st, you can tell the spider what to index, and what not to index. Craig. -----Original Message----- From: Jim McAtee [mailto:[EMAIL PROTECTED]] Sent: 11 December 2001 00:20 To: CF-Talk Subject: Re: OT: Keep search engine spyders from following links Joe, I'll try it out. Since the links all appear on CF-generated pages (pulled from database tables) I think I should be able to just Unicode the links within the CF templates themselves. Is there a CF Unicode tag or function somewhere, or do Unicode'd string simply use the ASCII codes for each character? Jim ----- Original Message ----- From: "Joseph DeVore" <[EMAIL PROTECTED]> To: "CF-Talk" <[EMAIL PROTECTED]> Sent: Monday, December 10, 2001 5:05 PM Subject: RE: OT: Keep search engine spyders from following links > Jim, > > It is possible to UNICODE HREFs and a lot of bots ignore these types of > links. I use unicoded HREFs for all of my email addresses so spam bots don't > grab them. I haven't tested unicoded HREFs in all browsers, just Netscape > 4.08 and IE 4+ on Windows > > If you want to take a look at a unicoded href you can use this form to > generate them: > > http://cftags.veloxweb.com/unicode/ > > > HTH, > > Joseph DeVore > VeloxWeb Technologies > > > > -----Original Message----- > From: Jim McAtee [mailto:[EMAIL PROTECTED]] > Sent: Monday, December 10, 2001 3:35 PM > To: CF-Talk > Subject: Re: OT: Keep search engine spyders from following links > > > IIS. What exactly would the ISAPI application do? > > I suppose within my redirection template I _could_ parse the cgi.useragent > string and neither record nor redirect known spyders. Keeping an up-to-date > list of agents might be a pain, though. I'd rather just tell the syders > (even if only the well-behaved ones) not to follow the link. > > Jim > > > ----- Original Message ----- > From: "Alex" <[EMAIL PROTECTED]> > To: "CF-Talk" <[EMAIL PROTECTED]> > Sent: Monday, December 10, 2001 3:39 PM > Subject: Re: OT: Keep search engine spyders from following links > > > > what's your web server? You can write an apache module to do this and I > > would assume you could do the same with ISAPI > > > > On Mon, 10 Dec 2001, Jim McAtee wrote: > > > > > On a couple of web pages, I run links to outside web sites through a CF > > > template that records the click-through along with some browser and > referer > > > stats in a table, then redirects the browser to the target URL. Is > there > > > any way to prevent search engine spyders from following these links? If > I'm > > > not mistaken, there's no way to use a robots.txt to prevent this > behavior, > > > or is there? > > > > > > Thanks, > > > Jim > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Get the mailserver that powers this list at http://www.coolfusion.com FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/[email protected]/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
