Not all robot obey robots.txt (although they should), but it's also quite possible you have made a mistake.
If the full path to your shopping cart is http://sub.domain.com/rel-path/shopping.asp then http://sub.domain.com/robots.txt should contain User-agent: * Disallow: /rel-path/shopping.asp Check the details very carefully. For example: - The location of the robots.txt file (at the root of the subdomain) - The disallow line path should always begin with a "/" - No blank line between "User-agent" and "Disallow" If that doesn't fix it, post your URL and someone will check it out for you I'm sure. :-) Alan Perkins e-Brand Management Limited > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of VBCoder > Sent: 01 December 2003 00:47 > To: [EMAIL PROTECTED] > Subject: [Robots] robots.txt questions > > > Hi, > I got to this list from robotstxt.org. From the looks of the > archive, most > here are building robots. My question is from the other > side, having robots > visit my site. I hope that my question can be answered here. > I have a site > that utilizes a shopping cart. The page to add goes like > shoppingcart.asp?item=add&item=w123456. My robots.txt file > has and entry > that ends with shoppingcart.asp. I see many robots that > visit the site, > read the robots.txt file and go merrily on their way to add > every item in > the catalog to the shopping cart. I have had to resort to > keeping records > of all of the UAs that look at the robots.txt file and adding > them to a list > that will return a 404 error if they try to add items to the > cart. I don't > see this as a real solution due to the overhead involved. It > does keep the > items out of the shopping cart database but the overhead cost > is too much. > The site in question is a sub web. The robots file is in the > domain root. > What am I doing wrong? > Thanks, > Paul > > > _______________________________________________ > Robots mailing list > [EMAIL PROTECTED] > http://www.mccmedia.com/mailman/listinfo/robots > _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
