Hello, I have a case where I need to crawl a list of exact url's. Somewhere in the range of 1 to 1.5M urls.
I have written those urls in numereus files under /home/urls , ie /home/urls/1 /home/urls/2 Then by using the crawl command I am crawling to depth=1 Are there any recomendations or general guidelines that I should follow when making nutch just to fetch and index a list of urls? Best Regards, C.B.
