Hi,
All of a sudden we started to get loads of errors related to browse indexes.
It looks like some robot is using http://qmro.qmul.ac.uk/jspui/browse url
while it should be using this url with some parameter.
****
An internal server error occurred on http://qmro.qmul.ac.uk/jspui:
Date: 4/24/13 2:38 PM
Session ID: 76F88FEE407DA917F0BAAE58640E8063
User: Anonymous
IP address: 138.37.31.252
-- URL Was: http://qmro.qmul.ac.uk/jspui/browse
-- Method: GET
-- Parameters were:
Exception:
javax.servlet.ServletException: There is no browse index for the request
*****
Is there any way to stop robot to browse the website?
We got 2 robots.txt
One is under /tomcat/webapps/jspui and has following content:
User-agent: *
# Uncomment the following line ONLY if sitemaps.org or HTML sitemaps are used
# and you have verified that your site is being indexed correctly.
# Disallow: /browse
Another robot.txt is at the root of apache /var/www/html and has following
content:
User-agent: *
Disallow: /community-list
Disallow: /simple-search
Disallow: /advanced-search
Disallow: /displaystats
Disallow: /subscribe
Disallow: /register
Disallow: /password-login
Sitemap: https://qmro.qmul.ac.uk/jspui/sitemap?map=0
Does adding ‘Disallow: /browse to’ any of these files will stop robot to
browse our repository?
Regards
Kirti
------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
