I am sory for your bandwidth consumption but I do not think that is my
spider. You are right about robot.txt files. I did not read them because I
do not know existance of that file. Thank you for advice. But I never send
my spider to your domain. I crawled Only small amount of host and none of
them about your host. How can you be sure about that is my spider. Please
warn me if the problem continues and you sure about the spider. 

Thank you

-----Original Message-----
From: bluebrit [mailto:[EMAIL PROTECTED] 
Sent: Monday, November 26, 2007 12:14 PM
To: nutch-agent@lucene.apache.org
Subject: Fw: Blocked nutch spider accessing pages
Importance: High

I sent the below original email to you without reply two weeks ago and as
you can see my domain is still being crawled by your spider.
Please advise me how to block it permanently from my domain or i will seek
avenues to report your spider for its intrusive behaviour to the major
search engines possibly resulting in your domains removal from their

      Nutch 1733+29 10.79 MB 23 Nov 2007 - 11:06 

owner blue-candy.com

----- Original Message ----- 
From: bluebrit 
To: nutch-agent@lucene.apache.org 
Sent: Monday, November 12, 2007 12:43 PM
Subject: Blocked nutch spider accessing pages

Hello, I am writing this email to you because of the following.

Blocked spider in robots.txt found in log file.

User-agent: Nutch
Disallow: /

To date this month Nutch has appeared in my site log an unreasonable amount
of times bearing in mind it is supposed to be blocked. It is obvious that
your spider is not reading the robots.txt file and as my domain contains a
copyright warning, can i assume you will be able to ensure that your spider
or the user of your spider will stop the repeated visits and possible
copying of text / graphics as well.

Below is a copy of log files from the last six months, that although not
large in bandwidth usage, does constitute a problem as it seems to show an
increasing demand.

      NutchCVS 588+8 2.37 MB 28 Jun 2007 - 06:43 

      Nutch 807+21 2.25 MB 28 Jun 2007 - 15:46 

      Nutch 324+223 1.35 MB 31 Jul 2007 - 04:11 

      Nutch 105+18 657.46 KB 31 Aug 2007 - 18:41 

      NutchCVS 712+12 2.73 MB 15 Aug 2007 - 00:38 

      Nutch 42+13 315.86 KB 30 Sep 2007 - 04:34 

      Nutch 30+12 182.74 KB 24 Oct 2007 - 19:56 

      Nutch 977+15 6.87 MB 08 Nov 2007 - 22:41 

My domain is http://www.blue-candy.com

Please note this is an adult domain and ALL of the images / video clips are
also copyright protected by the sponsoring companies.

Thank you for your reply regarding the above and for any additional
information you can supply regarding steps that can be taken to block Nutch
once and for all from spidering my domain.

Owner blue-candy.com

Reply via email to