On 10:27:55 28/Sep , AJ Chen wrote:
> I started the crawler with about 2000 sites. The fetcher could achieve
> 7 pages/sec initially, but the performance gradually dropped to about 2
> pages/sec, sometimes even 0.5 pages/sec. The fetch list had 300k pages
> and I used 500 threads. What are the main causes of this slowing down?
I have the same problem; I've tried with different number of fetchers
(10,20,50,100,..), but the download rate always decrease sistematically,
page after page.
The machine is a p4 1.7, 768 MB ram, running debian on 2.6.12 kernel.
The bandwidth isn't a problem (10Mbit in and 10Mbit out), but I cannot
obtain a stable, and high, page/s rate.
I've also tried to change machine and kernel, but the problem still
remains.
Can you please give us some advice?
Thank you for your help,
Menoz
--
Free Software Enthusiast
Debian Powered Linux User #332564
http://menoz.homelinux.org
