Obviously, Web Experts have very bad UPload bandwidth.

Frankly, classic installation of Apache with 150 "connections" will fail
against 15 threads of Nutch, nothing related to a bandwidth, even if it
is 8Mbps/800kbps for home-based sites.

May be Web Experts need to tune Apache Web Server, and use "worker"
model instead of "pre-fork"? It allows to handle 6000 concurrent users
(1024 RAM)... It saves memory using threads instead of processes...

-----Original Message-----
From: WebExpertsAmerica [mailto:[EMAIL PROTECTED] 
Sent: Friday, September 23, 2005 3:26 PM
Cc: nutch-agent@lucene.apache.org
Subject: Your Nutch Crawler is Out of Control - Apache Notified
Importance: High

You crawler is ignoring our robots.txt file.

http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)"

You are eating bandwidth at our domain in incredible amounts. This is

Please stop or we will be forced to block your IP and the crawler you
are using.

Best Regards,

Web Experts America

Whole Lot More for a Whole Lot LessC
$6/hr Professional Web Services http://www.WebExpertsAmerica.com


Website Solutions: http://www.WebExpertsAmerica.com/services.htm

AOL, MSN (Hotmail), and Yahoo
*Contact us anytime via chat. However, we DENY, BLOCK, and BAN anyone
that adds us to their Friend/Buddy list. Nothing personal, a security
policy to protect our chat connectivity from competitor abuse.

Terms of Service:

The information contained in this message is privileged and confidential
and protected from disclosure. If the reader of this message is not the
intended recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is strictly prohibited. If
you have received this communication in error, please notify us
immediately by replying to this message and then delete it from your

Reply via email to