Greetings list,
When Nutch (trunk) fetches using mapred, it seems to be assigning one
fetch thread (one slave node) to process previously-errored pages. Is
this as designed or do I have some type of malfunction?
When I perform a search large enough to observe the fetch process for
an extended period of time (1M pages over 16 nodes, in this case), I
notice there is one map task which performs _very_ poorly compared to
the others:
4905 pages, 33094 errors, 3.5 pages/s, 432 kb/s,
versus
46639 pages, 13227 errors, 43.9 pages/s, 4547 kb/s,
It is deficient in terms of raw pages/sec, execution time (it is the
last map task to complete), and the number of errors encountered.
As I said, there seems to always be exactly one map task like this.
Different fetch executions will have the thread assigned to different
machines -- there doesn't seem to be any pattern.
What the heck is going on here?
-Shawn
