Doğacan Güney wrote:
Hi everyone,
Has anyone tried Fetcher2 from latest trunk? On our tests, Fetcher2 is
always slower (by a large margin) that Fetcher.
For a segment with ~30000 urls, we ran Fetcher with 150 threads and
Fetcher2 with 50 threads. Fetcher finishes around 1 hour, while
Fetcher2 takes around 4 hours. We ran this test more than once and
got similar results.
Are we running Fetcher2 with too few/too many threads? I was under the
impression that Fetcher2 doesn't need as many threads as Fetcher since
threads do not block.
Yes, that was the idea. Could you test it with the same number of
threads? Is the configuration identical in all other aspects?
Are you running the version with the fix from NUTCH-474?
Any suggestions?
If you already have a setup to reproduce this, you could perhaps spend
some time debugging this ... add some timing info, and queue info logging.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
