I'm currently fetching with 35 threads. The CPU load is about 5-10% (P4 3.0 HT). Parsing obviously isn't using many resources.
Removing parsing also would not speed up the fetching process. If parsing (while fetching) is removed (with a command line argument), I'll probably tune the fetcher down to 30 threads and have the same overall fetching speed. -----Original Message----- From: Peter Veentjer - Anchor Men [mailto:[EMAIL PROTECTED] Sent: Monday, September 05, 2005 6:01 AM To: [email protected] Subject: fetcher question: why multithreaded? Hi, I`m looking at the code of the fetcher and have the following question: why does the fetcher do more than fetching? Wouldn`t it be better te move the page parsing to another component and let the fetcher only fetch? (so the fetch threads only do fetching). Another problem with this threaded approach is that you need a lot of threads because a single thread is responsible for retrieving data and also for parsing it. If you remove the parsing part, a thread would only be responsible for fetching. And this makes it possible to use a single thread in the Fetcher that gathers data from a lot of sockets (and this reduces context switching overhead). This is a technique widely used in search engines and I`m curious about why Nutch goes for a different approach. Met vriendelijke groet, Peter Veentjer Anchor Men Interactive Solutions - duidelijk in zakelijke internetoplossingen Praediniussingel 41 9711 AE Groningen T: 050-3115222 F: 050-5891696 E: [EMAIL PROTECTED] I : www.anchormen.nl <blocked::http://www.anchormen.nl/> ------------------------------------------------------- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
