Dear Nutchers,
I noticed same problem twise, with PentiumMobile2Mhz & WindowsXP & 2Gb, and with 2xOpteron252 x SuseLinux x 4Gb I have only one explanation which should be probably mirrored at JIRA: ================ Network. ======== 1. I never had such a problem with The Grinder, http://grinder.sourceforge.net, which is based on alternate HTTPClient http://www.innovation.ch/java/HTTPClient/index.html. Apache SF should really review their HttpClient RC3(!!!) accordingly, HTTPClient (upper--HTTP-case)is not "alpha", it is production version... I used Grinder a lot, it allows to execute 32 processes with 64 threads each on 2048Mb RAM... 2. I found at SUN API this: java.net.Socket public void setReuseAddress(boolean on) - please check API!!! 3. I saw in your PROTOCOL-HTTP this code: ... HTTP/1.0 ... Why? Why version 1.0??? It should understand server's reply such as "Connection: close" "Connection: keep-alive" etc. (pls ignore typo). 4. By the way, how many files UNIX needs in order to maintain 65536 network sockets? Respectfully, Fuad P.S. Sorry guys, I don't have anough time to participate... Could you please test this suspicious behaviour, and very strange opinion? Should I create a new bug report at JIRA? SUN's Socket, Apache's HttpClient, UNIX's networking... -----Original Message----- From: Daniele Menozzi [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 28, 2005 4:42 PM To: [email protected] Subject: Re: what contibute to fetch slowing down On 10:27:55 28/Sep , AJ Chen wrote: > I started the crawler with about 2000 sites. The fetcher could > achieve > 7 pages/sec initially, but the performance gradually dropped to about 2 > pages/sec, sometimes even 0.5 pages/sec. The fetch list had 300k pages > and I used 500 threads. What are the main causes of this slowing down? I have the same problem; I've tried with different number of fetchers (10,20,50,100,..), but the download rate always decrease sistematically, page after page. The machine is a p4 1.7, 768 MB ram, running debian on 2.6.12 kernel. The bandwidth isn't a problem (10Mbit in and 10Mbit out), but I cannot obtain a stable, and high, page/s rate. I've also tried to change machine and kernel, but the problem still remains. Can you please give us some advice? Thank you for your help, Menoz -- Free Software Enthusiast Debian Powered Linux User #332564 http://menoz.homelinux.org
