I believe that "redirect" is the important phrase here. Are you
talking an actual meta redirect? or a branch. If using a meta
redirect, the robot will only follow a 301 permanent redirect,
otherwise it stops.
I have a routine that every page in my site executes, so implementation
should be pretty easy.
So every time a visitor selects a page, I check for a userid. If not present
I would check for a cookie. If neither, I would redirect to the "login
benefits" page.
If I don't allow spiders to crawl this page (with robots.txt) and this is
the page the visitor is redirected to if they don't have a cookie or userid,
wouldn't that stop the spider from crawling the site?
Steve
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 18, 2004 9:09 AM
To: [EMAIL PROTECTED]
Subject: Re: Witango-Talk: Login/Affiliates/Cookies/Spiders (OT)
Create a robots.txt file and add it to the top of the heiarchy of the
site. In the robots.txt file, add the following....
User-agent: *
Disallow: DoNotCrawlThisPage.taf
The DoNotCrawlThisPage.taf is the page you do not want robots to crawl.
Hope this helps
Fogelson, Steve wrote:
Windows 2003 Web Edition IIS 6.0, R:Tango 5, Oterro 3.0
I would like to suggest that visitors login to a site when they first
enter
it. Presently I check to see if they have a user scoped userid. If they
don't I set it to the session cookie userreference. So I was thinking that
if they didn't have a user id, I would redirect to a page that would list
benefits of signing in and provide login input fields.
I also want to create a persistent cookie to place on their computer that
would contain an affiliate code if the visitor was referred to the site by
an affiliate. So I could check to see if they have this cookie first. If
they have a cookie, it would be nice to log them in as well as read their
affiliate code. But I don't know how to transmit encrypted account and
password info in the form of a cookie first to their computer and then at
a
later date (subsequent visits) back to my site. Any comments?
I also have a challenge when a search engine spider hits the site. I don't
want to redirect the spider to a "login suggestion" page. Any suggestions
on
how to determine if the visitor is a spider? Maybe use <@CGIPARAM>. I
noticed when looking at my IIS logs, that there are entries like the
following:
Why do you care if the spider indexes your login suggestion page.
Assuming that page still has links to the content that you want to
be indexed by the spider, I don't see any harm in it indexing that
suggestion page. And if you didn't want it to index that page,
there might even be an http heaeder or <META> tag that you could add
to just that login suggestion page that would cause the bot to not
index it.
To answer your question below, you can get to that data through the
@CGIPARAM tag. It's the "USER_AGENT"
/John
66.196.93.29 HTTP/1.0
YahooSeeker/1.1+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://help.yahoo.com/
he
lp/us/shop/merchant/)
64.68.82.55 HTTP/1.0 Googlebot/2.1+(+http://www.googlebot.com/bot.html)
66.196.65.40 HTTP/1.0
Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysear
ch
/slurp)
Is this accessible by Witango? Is it consistent? Is it useful?
Thanks for any ideas.
Steve Fogelson
Internet Commerce Solutions
________________________________________________________________________
TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
________________________________________________________________________
TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
--
________________________________________________________________________
TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
________________________________________________________________________
TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
--
________________________________________________________________________
TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
