I was saying that based on what the previous poster stated. Also the fact that I have read through quite a bit of posts stating that the problem with crawling in a vertical environment has to do with the way fetcher2 was built. The fetches are grouped by domain name and if you have a lot of urls from the same domain then you are not able to do quick mapreduce jobs.
I hope this is wrong though ;-) -- View this message in context: http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2932969.html Sent from the Nutch - User mailing list archive at Nabble.com.
