Hi, I do actually return a "403 Search Engines are forbidden to add items to the shopping cart" response, though I don't think they get the text part. I would fear that the search engines would index it if I added any of the other text you suggested.
Yes the ones I am referring to fetch robots.txt. The domain is www.coseco.com the sub web is heartnart. The URL would be http://www.coseco.com/heartnart/shoppingcart.asp and a robots.txt file is in both the root and the sub web and some robots actually look at the one in the sub web. I added it there to see if it resolved this but it didn't. the root one was there alone for several months before I added the one in the sub web. Please have a look and let me know if I am doing something wrong or if I should add something I am not. Paul -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Klaus Johannes Rusch Sent: Monday, December 01, 2003 4:33 AM To: Internet robots, spiders, web-walkers, etc.; [EMAIL PROTECTED] Subject: Re: [Robots] robots.txt questions VBCoder wrote: > I got to this list from robotstxt.org. From the looks of the archive, most > here are building robots. My question is from the other side, having robots > visit my site. I hope that my question can be answered here. I have a site > that utilizes a shopping cart. The page to add goes like > shoppingcart.asp?item=add&item=w123456. My robots.txt file has and entry > that ends with shoppingcart.asp. I see many robots that visit the site, > read the robots.txt file and go merrily on their way to add every item in > the catalog to the shopping cart. I have had to resort to keeping records > of all of the UAs that look at the robots.txt file and adding them to a list > that will return a 404 error if they try to add items to the cart. I'd recommend to return a 403 Forbidden or, probably even better, redirect all such robots to an explanatory page with your contact information, in case you incorrectly identify some requests as robot traffic and a human being ends up on that page. > I don't > see this as a real solution due to the overhead involved. It does keep the > items out of the shopping cart database but the overhead cost is too much. > The site in question is a sub web. The robots file is in the domain root. > What am I doing wrong? Do these robots which crawl your site actually fetch robots.txt? Not all robots honor robots.txt, if your logs show that they did fetch robots.txt there may be a problem with your robots file, feel free to post the URL here or in private email if you need a second pair of eyes to look over it. -- Klaus Johannes Rusch [EMAIL PROTECTED] http://www.atmedia.net/KlausRusch/ _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots --- Incoming mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.545 / Virus Database: 339 - Release Date: 11/27/2003 _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
