Hi there, I just started working on a search engine based on the nutch
project, but we are finding that the fetcher is crawling extremely slow.
I've seen posts talking about people maxing out their 5mb lines with the
fetcher, but we can't seem to get anymore than about 20k/s or 1.5
pages/second, which isnt even a smidgen of our capacity, even with
-threads set to 200 . This is using the mapred branch, in freebsd 4.
Are there any settings we might be missing that would cause this
slowdown? or are there certain network configurations that could be
causing this?
Also, is the -numFetchers option in 'nutch generate' broken in the
mapred branch? it worked fine in 0.7, but doesn't seem to do anything in
0.8-dev.
Thanks a lot for your help.
Matt Zytaruk
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
