Re: [Dspace-tech] Issue about Google crawler

2010-10-05 Thread Kim Shepherd
Hi Panyarak, It might be an idea to add /displaystats to your JSPUI's robots.txt and to any Google Webmaster Tools robots.txt files or Page Removal Requests. For Google to de-index pages, it generally likes to see a 404 (not found) or a 410 (gone). Unfortunately, the servlet that handles

Re: [Dspace-tech] Issue about Google crawler

2010-10-05 Thread Kim Shepherd
I should point out that my robots.txt suggestions assume you don't want any stats pages crawled at all... if that's not true, it's probably best to apply the patch for DS-689 and wait for Google to de-index (and make the robots.txt entries more specific if there are only a few invalid handles