I use nutch 1.0 to crawl 10M URLs,I found the fetch slow down when the
urls below almost 3000 or 4000 urls. and I change the threads to different
numbers(1800,1000,600),but make no difference.
     That look like, at the begining or when the urls number is large ,the
speed is very fast.but in the en more and more slow.

Best wishes!

Reply via email to