VBCoder wrote: > I got to this list from robotstxt.org. From the looks of the archive, most > here are building robots. My question is from the other side, having robots > visit my site. I hope that my question can be answered here. I have a site > that utilizes a shopping cart. The page to add goes like > shoppingcart.asp?item=add&item=w123456. My robots.txt file has and entry > that ends with shoppingcart.asp. I see many robots that visit the site, > read the robots.txt file and go merrily on their way to add every item in > the catalog to the shopping cart. I have had to resort to keeping records > of all of the UAs that look at the robots.txt file and adding them to a list > that will return a 404 error if they try to add items to the cart.
I'd recommend to return a 403 Forbidden or, probably even better, redirect all such robots to an explanatory page with your contact information, in case you incorrectly identify some requests as robot traffic and a human being ends up on that page. > I don't > see this as a real solution due to the overhead involved. It does keep the > items out of the shopping cart database but the overhead cost is too much. > The site in question is a sub web. The robots file is in the domain root. > What am I doing wrong? Do these robots which crawl your site actually fetch robots.txt? Not all robots honor robots.txt, if your logs show that they did fetch robots.txt there may be a problem with your robots file, feel free to post the URL here or in private email if you need a second pair of eyes to look over it. -- Klaus Johannes Rusch [EMAIL PROTECTED] http://www.atmedia.net/KlausRusch/ _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
