VBCoder wrote: > I do actually return a "403 Search Engines are forbidden to add items to the > shopping cart" response, though I don't think they get the text part. I > would fear that the search engines would index it if I added any of the > other text you suggested.
Search engines generally do not store pages with a 4xx status code, so as long as your status code is 403 you can safely place explanatory text on the homepage. > Yes the ones I am referring to fetch robots.txt. The domain is > www.coseco.com the sub web is heartnart. The URL would be > http://www.coseco.com/heartnart/shoppingcart.asp and a robots.txt file is in > both the root and the sub web and some robots actually look at the one in > the sub web. I added it there to see if it resolved this but it didn't. > the root one was there alone for several months before I added the one in > the sub web. Please have a look and let me know if I am doing something > wrong or if I should add something I am not. Your server seems to be case-insensitive, and www.heartnart.com redirects to http://www.coseco.com/HeartnArt/shoppingcart.asp -- note that HeartnArt is spelled in uppercase, whereas shoppingcart.asp is all lowercase, whereas you have it in mixed case in your robots.txt file. The best solution would be translating all requests to lowercase, which makes maintaining the robots.txt file a lot easier. Alternatively you can try excluding all combinations you don't want indexed, although there may be more than you want to add to robots.txt. -- Klaus Johannes Rusch [EMAIL PROTECTED] http://www.atmedia.net/KlausRusch/ _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
