Author: Prateep Email: Message: Hi,
I have a number of user submitted URLs that I want to index without following. It works fine for the most part, but when I encounter directory URLs that return a 301, the page specified in the redirect is not being indexed. For example, I have the following URLs: http://www.foo.com/bar http://www.blah.com/xxx bar and xxx are directories, so the server sends a 301 redirect response to point to these: http://www.foo.com/bar/ http://www.blah.com/xxx/ The relevant section in my indexer.conf includes a dynamically generated file which contains the following: Robots no Follow no Server page http://www.foo.com/bar Server page http://www.blah.com/xxx My question is, is there an easy way to get it to index the result of a 301 (and that page only). I guess an alternative is to append a "/" to all URLs that look like directories but I have quite a few and would like it to work if the URL correctly ends in a "/" or not. Hoping someone can help... Prateep Reply: <http://www.mnogosearch.org/board/message.php?id=4705> ___________________________________________ If you want to unsubscribe send "unsubscribe general" to [EMAIL PROTECTED]
