Hi, Every place I have read about robots.txt rules state that it is supposed to be case insensitive. You seem to be suggesting that this is wrong. I have added lines the include the exact case of the offender, but this does not seem to stop them. The mixed case lines are and experiment, the all lower case lines should be enough to stop them from what I have read. Are you suggesting that robots.txt needs to be case sensitive? The domain name heartnart.com does a redirect to www.coseco.com/heartnart . I would think that the case should not matter to a search engine as it doesn't to the web in general. I mix them so that it can be more easily read by a humans. Are you suggesting that a search engine would think that www.coseco.com/heartnart is a different place than www.coseco.com/HeartnArt? I am more confused than before.
Paul -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Klaus Johannes Rusch Sent: Monday, December 01, 2003 7:54 AM To: VBCoder Cc: Internet robots, spiders, web-walkers, etc. Subject: Re: [Robots] robots.txt questions VBCoder wrote: > I do actually return a "403 Search Engines are forbidden to add items to the > shopping cart" response, though I don't think they get the text part. I > would fear that the search engines would index it if I added any of the > other text you suggested. Search engines generally do not store pages with a 4xx status code, so as long as your status code is 403 you can safely place explanatory text on the homepage. > Yes the ones I am referring to fetch robots.txt. The domain is > www.coseco.com the sub web is heartnart. The URL would be > http://www.coseco.com/heartnart/shoppingcart.asp and a robots.txt file is in > both the root and the sub web and some robots actually look at the one in > the sub web. I added it there to see if it resolved this but it didn't. > the root one was there alone for several months before I added the one in > the sub web. Please have a look and let me know if I am doing something > wrong or if I should add something I am not. Your server seems to be case-insensitive, and www.heartnart.com redirects to http://www.coseco.com/HeartnArt/shoppingcart.asp -- note that HeartnArt is spelled in uppercase, whereas shoppingcart.asp is all lowercase, whereas you have it in mixed case in your robots.txt file. The best solution would be translating all requests to lowercase, which makes maintaining the robots.txt file a lot easier. Alternatively you can try excluding all combinations you don't want indexed, although there may be more than you want to add to robots.txt. -- Klaus Johannes Rusch [EMAIL PROTECTED] http://www.atmedia.net/KlausRusch/ _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots --- Incoming mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.545 / Virus Database: 339 - Release Date: 11/27/2003 _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
