how fast could you fetch with nutch?We fetch with around 13 pages/s and around 1500 kb/s at the beginning. Then it is slowing down a litte bit.
33 ++ pages/s
What do you think is normally the bottleneck?
Threads (not to much) A well mixed fetchlist, not to much urls from the same hosts. We notice a local DNS is important. Stefan
