I have a routine that every page in my site executes, so implementation should be pretty easy.
So every time a visitor selects a page, I check for a userid. If not present I would check for a cookie. If neither, I would redirect to the "login benefits" page. If I don't allow spiders to crawl this page (with robots.txt) and this is the page the visitor is redirected to if they don't have a cookie or userid, wouldn't that stop the spider from crawling the site? Steve -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Thursday, March 18, 2004 9:09 AM To: [EMAIL PROTECTED] Subject: Re: Witango-Talk: Login/Affiliates/Cookies/Spiders (OT) Create a robots.txt file and add it to the top of the heiarchy of the site. In the robots.txt file, add the following.... User-agent: * Disallow: DoNotCrawlThisPage.taf The DoNotCrawlThisPage.taf is the page you do not want robots to crawl. Hope this helps >Fogelson, Steve wrote: > >>Windows 2003 Web Edition IIS 6.0, R:Tango 5, Oterro 3.0 >> >>I would like to suggest that visitors login to a site when they first enter >>it. Presently I check to see if they have a user scoped userid. If they >>don't I set it to the session cookie userreference. So I was thinking that >>if they didn't have a user id, I would redirect to a page that would list >>benefits of signing in and provide login input fields. >>I also want to create a persistent cookie to place on their computer that >>would contain an affiliate code if the visitor was referred to the site by >>an affiliate. So I could check to see if they have this cookie first. If >>they have a cookie, it would be nice to log them in as well as read their >>affiliate code. But I don't know how to transmit encrypted account and >>password info in the form of a cookie first to their computer and then at a >>later date (subsequent visits) back to my site. Any comments? >> >>I also have a challenge when a search engine spider hits the site. I don't >>want to redirect the spider to a "login suggestion" page. Any suggestions on >>how to determine if the visitor is a spider? Maybe use <@CGIPARAM>. I >>noticed when looking at my IIS logs, that there are entries like the >>following: >> >Why do you care if the spider indexes your login suggestion page. >Assuming that page still has links to the content that you want to >be indexed by the spider, I don't see any harm in it indexing that >suggestion page. And if you didn't want it to index that page, >there might even be an http heaeder or <META> tag that you could add >to just that login suggestion page that would cause the bot to not >index it. > >To answer your question below, you can get to that data through the >@CGIPARAM tag. It's the "USER_AGENT" > >/John > >>66.196.93.29 HTTP/1.0 >>YahooSeeker/1.1+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://help.yahoo.com/ he >>lp/us/shop/merchant/) >>64.68.82.55 HTTP/1.0 Googlebot/2.1+(+http://www.googlebot.com/bot.html) >>66.196.65.40 HTTP/1.0 >>Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysear ch >>/slurp) >> >>Is this accessible by Witango? Is it consistent? Is it useful? >> >>Thanks for any ideas. >>Steve Fogelson >>Internet Commerce Solutions >>________________________________________________________________________ >>TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf >> >> > >________________________________________________________________________ >TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf -- ________________________________________________________________________ TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf ________________________________________________________________________ TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
