On Sun, 30 Nov 2003, VBCoder wrote:
> that utilizes a shopping cart. The page to add goes like
> shoppingcart.asp?item=add&item=w123456. My robots.txt file has and entry
> that ends with shoppingcart.asp. I see many robots that visit the site,
Way back at the beginning of time (1996 or so) I t
L PROTECTED]
Behalf Of Nick Arnett
Sent: Monday, December 01, 2003 10:33 AM
To: Internet robots, spiders, web-walkers, etc.
Subject: Re: [Robots] robots.txt questions
VBCoder wrote:
> Hi,
> Every place I have read about robots.txt rules state that it is supposed
to
> be case insensitive.
VBCoder wrote:
Hi,
Every place I have read about robots.txt rules state that it is supposed to
be case insensitive.
The spec says "A case insensitive substring match of the name without
version information is recommended." This is up to the robots, not you.
You probably are getting hit by rob
ECTED] [mailto:[EMAIL PROTECTED]
Behalf Of Klaus Johannes Rusch
Sent: Monday, December 01, 2003 7:54 AM
To: VBCoder
Cc: Internet robots, spiders, web-walkers, etc.
Subject: Re: [Robots] robots.txt questions
VBCoder wrote:
> I do actually return a "403 Search Engines are forbidden to add
VBCoder wrote:
> I do actually return a "403 Search Engines are forbidden to add items to the
> shopping cart" response, though I don't think they get the text part. I
> would fear that the search engines would index it if I added any of the
> other text you suggested.
Search engines generally d
TED]
Behalf Of Klaus Johannes Rusch
Sent: Monday, December 01, 2003 4:33 AM
To: Internet robots, spiders, web-walkers, etc.; [EMAIL PROTECTED]
Subject: Re: [Robots] robots.txt questions
VBCoder wrote:
> I got to this list from robotstxt.org. From the looks of the archive,
most
> here are buildi
VBCoder wrote:
> I got to this list from robotstxt.org. From the looks of the archive, most
> here are building robots. My question is from the other side, having robots
> visit my site. I hope that my question can be answered here. I have a site
> that utilizes a shopping cart. The page to a
Not all robot obey robots.txt (although they should), but it's also
quite possible you have made a mistake.
If the full path to your shopping cart is
http://sub.domain.com/rel-path/shopping.asp
then http://sub.domain.com/robots.txt should contain
User-agent: *
Disallow: /rel-path/shopping.asp
