On Sun, 30 Nov 2003, VBCoder wrote:
that utilizes a shopping cart. The page to add goes like
shoppingcart.asp?item=additem=w123456. My robots.txt file has and entry
that ends with shoppingcart.asp. I see many robots that visit the site,
Way back at the beginning of time (1996 or so) I
: Monday, December 01, 2003 10:33 AM
To: Internet robots, spiders, web-walkers, etc.
Subject: Re: [Robots] robots.txt questions
VBCoder wrote:
Hi,
Every place I have read about robots.txt rules state that it is supposed
to
be case insensitive.
The spec says A case insensitive substring match
