RP wrote:
Hello -
Decided to give the fetch2 a workout from a recent nightly build in
the 9.x trunk. Very slow - lots of "spinwaiting" messages (from 2 to
10) in between every fetch in the hadoop.log. I've tried threads at
100 and 500 and same behavior is seen so in effect the fetch rate is
way below 1 page/sec when I do this. Same crawl list with old fetch
flies by comparison ~ 10pages/sec.... Box is only doing nutch. Anyone
else seen this or have a workaround..??
Looks weird. So far I've been getting a similar or better performance
from fetcher2.
If you are crawling public URLs, could you please send me your fetchlist ?
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
