Hi to all, I'm a novice user of Nutch, I have it on a debian machine, and I have probe the latest release of nutch 1.0 with very slow results in crawling, I have a 10 Megabytes/s connection and it only crawls at 300 Kb/s with peaks of 1 Mb/s. I tweaked everything, dns, linux tcp settings, thread numbers, java conf etc.. but anything have effect. There are a lot of spin waiting threads there, only 10-20 of them working and I have injected 1M different hosts, so politeness is not the problem. I swithched back to 0.9 nutch, and then it works like a charm at good speeds 5-6 Mb/s with the bottleneck on machine cpu.
¿Are this issues solved on dev version of nutch or why is this happens? Thanks in advance, -- View this message in context: http://old.nabble.com/Nutch-1.0-slow-crawls-tp27243302p27243302.html Sent from the Nutch - User mailing list archive at Nabble.com.
