I've got another issue. I'm crawling a single domain and I have fetcher.queue.mode set to "byHost". I'm using 20 thread so I set fetcher.threads.per.queue to 20. But I get a NullPointerException.
But setting it to 10 thread, it works fine. Can someone explain me why nutch has this behavior? Sorry tfor bothering you and thanks very much for your help. Matteo 2012/9/11 Markus Jelsma <[email protected]> > Hi > > -----Original message----- > > From:Matteo Simoncini <[email protected]> > > Sent: Tue 11-Sep-2012 14:41 > > To: [email protected] > > Subject: Parallelize Fetching Phase > > > > Hi everyone, > > > > I'm running nutch 1.5.1 using a script I created, but there is > > a significant slowdown in the fetching phase. > > My script uses 20 thread to fetch. Here is the fetch istruction: > > > > bin/nutch fetch $segment -threads 20 > > > > It works, but it seems they are all fetching the same URL. Here is the > log: > > Not same URL but same host or domain. The fetcher uses either host, domain > or IP queues. If you have only one domain or host them setting > fetcher.queue.mode is useless. Instead, you would have to increase > fetcher.threads.per.queue. > > > > > > fetching > > > http://www.eclap.eu/drupal/?q=en-US/node/65229/og/forum&sort=asc&order=Topic > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > > fetching > > > http://www.eclap.eu/drupal/?q=en-US/node/2867/og/forum&sort=asc&order=Created > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > > fetching http://www.eclap.eu/drupal/?q=zh-hans/node/103996 > > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=412 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412 > > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412 > > ... > > > > Is there a way to make each thread crawl a different URL? > > >
