It makes no difference if I set fetcher.threads.per.host to 1 or 100, which
I assume is what you were suggesting?
I also stated that the majority of pages to fetch were from unique hosts, so
I believe the value of this parameter should not really come into play.
Cheers...
Roger
--------------------------------------------------
From: "Laurent Laborde" <kerdez...@gmail.com>
Sent: Tuesday, February 03, 2009 5:51 PM
To: <nutch-user@lucene.apache.org>
Subject: Re: Fetcher2 Slow
On Tue, Feb 3, 2009 at 4:10 AM, Roger Dunk <ro...@at.com.au> wrote:
Hi all,
I'm having no luck whatsoever using Fetcher2, as even with 50 threads
enabled and parsing disabled, I have 48 or 49 threads SpinWaiting, and 0
hosts in the queue. I do however have some 50,000 pages to fetch, the
majority of which are from unique hosts.
The regular fetcher works as expected, fetching concurrently from 50
hosts.
There is a configuration parameters limiting the concurent fetcher per
unique host.
--
F4FQM
Kerunix Flan
Laurent Laborde
