Hi -----Original message----- > From:Matteo Simoncini <[email protected]> > Sent: Tue 11-Sep-2012 14:41 > To: [email protected] > Subject: Parallelize Fetching Phase > > Hi everyone, > > I'm running nutch 1.5.1 using a script I created, but there is > a significant slowdown in the fetching phase. > My script uses 20 thread to fetch. Here is the fetch istruction: > > bin/nutch fetch $segment -threads 20 > > It works, but it seems they are all fetching the same URL. Here is the log:
Not same URL but same host or domain. The fetcher uses either host, domain or IP queues. If you have only one domain or host them setting fetcher.queue.mode is useless. Instead, you would have to increase fetcher.threads.per.queue. > > fetching > http://www.eclap.eu/drupal/?q=en-US/node/65229/og/forum&sort=asc&order=Topic > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=414 > fetching > http://www.eclap.eu/drupal/?q=en-US/node/2867/og/forum&sort=asc&order=Created > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=413 > fetching http://www.eclap.eu/drupal/?q=zh-hans/node/103996 > -activeThreads=20, spinWaiting=19, fetchQueues.totalSize=412 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412 > -activeThreads=20, spinWaiting=20, fetchQueues.totalSize=412 > ... > > Is there a way to make each thread crawl a different URL? >
