Hello,
Following to some discussions, developpers mails, ... I tried to get the
best performances (pages/second) for the following case :
- 120 web servers to crawl
- 10 Mbits/s connexion
I reached about 3 Mbits/s average fetching speed with following
parameters (unpolite mode) :
- fetcher.server.delay = 1.0
- fetcher.per.host = 20
- threads = 800
- http.timeout = 5000
I see that Nutch is very slow for the first minuts ... performances
increase with time : it is now at 2500 kb/s and was at 2000kb/s 5
minutes ago.
segment 20050802115311, 7200 pages, 446 errors, 231654440 bytes, 706020 ms
050802 120623 148 status: 10.198011 pages/s, 2563.3838 kb/s, 32174.227
bytes/page
I read Doug Cutting mail about fetcher.max.delay, but i still don't
understand how i cannot reach 10 mbits/s speed with 120 different servers.
Any tips to increase my performances please ?
Thank you very much.
Christophe Noël
Cetic Grid Data Mining
