please share the relevant part of the log.
-----Original message-----
> From:Matteo Simoncini <[email protected]>
> Sent: Wed 12-Sep-2012 11:23
> To: [email protected]
> Subject: Re: Parallelize Fetching Phase
>
> I've got another issue.
>
> I'm crawling a single domain and I have fetcher.queue.mode set to "byHost".
> I'm using 20 thread so I set fetcher.threads.per.queue to 20. But I get a
> NullPointerException.
>
> But setting it to 10 thread, it works fine.
>
> Can someone explain me why nutch has this behavior?
>
> Sorry tfor bothering you and thanks very much for your help.
>
> Matteo
>
> 2012/9/11 Markus Jelsma <[email protected]>
>
> > Hi
> >
> > -----Original message-----
> > > From:Matteo Simoncini <[email protected]>
> > > Sent: Tue 11-Sep-2012 14:41
> > > To: [email protected]
> > > Subject: Parallelize Fetching Phase
> > >
> > > Hi everyone,
> > >
> > > I'm running nutch 1.5.1 using a script I created, but there is
> > > a significant slowdown in the fetching phase.
> > > My script uses 20 thread to fetch. Here is the fetch istruction:
> > >
> > > bin/nutch fetch $segment -threads 20
> > >
> > > It works, but it seems they are all fetching the same URL. Here is the
> > log:
> >
> > Not same URL but same host or domain. The fetcher uses either host, domain
> > or IP queues. If you have only one domain or host them setting
> > fetcher.queue.mode is useless. Instead, you would have to increase
> > fetcher.threads.per.queue.
> >
> >
> > >
> > > fetching
> > >
> > http://www.eclap.eu/drupal/?q=en-US/node/65229/og/forum&sort=asc&order=Topic
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414
> > > fetching
> > >
> > http://www.eclap.eu/drupal/?q=en-US/node/2867/og/forum&sort=asc&order=Created
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413
> > > fetching http://www.eclap.eu/drupal/?q=zh-hans/node/103996
> > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=412
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412
> > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412
> > > ...
> > >
> > > Is there a way to make each thread crawl a different URL?
> > >
> >
>
