I tried increasing the numbers of threads to 50 but the speed is not affected
I tried changing the partition.url.mode value to byDomain and fetcher.queue.mode to byDomain but still it does not help the speed. It seems to get urls from 2 domains now and the other domains are not getting crawled. Is this due to the url score? if so how do i crawl urls from all the domains? lewis john mcgibbney wrote > Increase number of threads when fetching > Also please see nutch-deault.xml for paritioning of urls, if you know your > target domains you may wish to adapt the policy. > Lewis > > On Sunday, January 27, 2013, peterbarretto < > peterbarretto08@ > > > wrote: >> I want to increase the number of urls fetched at a time in nutch. I have >> around 10 websites to crawl. so how can i crawl all the sites at a time ? >> right now i am fetching 1 site with a fetch delay of 2 second but it is > too >> slow. How to concurrently fetch from different domain? >> >> >> >> -- >> View this message in context: > http://lucene.472066.n3.nabble.com/increase-the-number-of-fetches-at-agiven-time-on-nutch-1-6-or-2-1-tp4036499.html >> Sent from the Nutch - User mailing list archive at Nabble.com. >> > > -- > *Lewis* -- View this message in context: http://lucene.472066.n3.nabble.com/increase-the-number-of-fetches-at-agiven-time-on-nutch-1-6-or-2-1-tp4036499p4036630.html Sent from the Nutch - User mailing list archive at Nabble.com.
