Hello, I am writing this email to you because of the following.
Blocked spider in robots.txt found in log file.
User-agent: Nutch
Disallow: /
To date this month Nutch has appeared in site log an unreasonable amount of
times bearing in mind it is supposed to be blocked. It is obvious that your
possibly resulting in your domains removal from their listings.
Nutch 1733+29 10.79 MB 23 Nov 2007 - 11:06
regards
owner blue-candy.com
- Original Message -
From: bluebrit
To: nutch-agent@lucene.apache.org
Sent: Monday, November 12, 2007 12:43 PM
Subject: Blocked nutch
