User-agent: * Disallow: DoNotCrawlThisPage.taf
The DoNotCrawlThisPage.taf is the page you do not want robots to crawl.
Hope this helps
Fogelson, Steve wrote:
Windows 2003 Web Edition IIS 6.0, R:Tango 5, Oterro 3.0Why do you care if the spider indexes your login suggestion page. Assuming that page still has links to the content that you want to be indexed by the spider, I don't see any harm in it indexing that suggestion page. And if you didn't want it to index that page, there might even be an http heaeder or <META> tag that you could add to just that login suggestion page that would cause the bot to not index it.
I would like to suggest that visitors login to a site when they first enter it. Presently I check to see if they have a user scoped userid. If they don't I set it to the session cookie userreference. So I was thinking that if they didn't have a user id, I would redirect to a page that would list benefits of signing in and provide login input fields. I also want to create a persistent cookie to place on their computer that would contain an affiliate code if the visitor was referred to the site by an affiliate. So I could check to see if they have this cookie first. If they have a cookie, it would be nice to log them in as well as read their affiliate code. But I don't know how to transmit encrypted account and password info in the form of a cookie first to their computer and then at a later date (subsequent visits) back to my site. Any comments?
I also have a challenge when a search engine spider hits the site. I don't want to redirect the spider to a "login suggestion" page. Any suggestions on how to determine if the visitor is a spider? Maybe use <@CGIPARAM>. I noticed when looking at my IIS logs, that there are entries like the following:
To answer your question below, you can get to that data through the @CGIPARAM tag. It's the "USER_AGENT"
/John
66.196.93.29 HTTP/1.0 YahooSeeker/1.1+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://help.yahoo.com/he lp/us/shop/merchant/) 64.68.82.55 HTTP/1.0 Googlebot/2.1+(+http://www.googlebot.com/bot.html) 66.196.65.40 HTTP/1.0 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch /slurp)
Is this accessible by Witango? Is it consistent? Is it useful?
Thanks for any ideas. Steve Fogelson Internet Commerce Solutions ________________________________________________________________________ TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
________________________________________________________________________ TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
-- ________________________________________________________________________ TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
