Re: [Nutch-dev] multiple crawlers

[Doug Cutting]

Yes.  Use the -numFetchers option to the 'generate' command to generate 
multiple fetchlists that can be fetched in parallel.  Fetchlists 
generated this way have disjoint sets of hosts, so that politeness can 
be enforced.  The 'updatedb' command also accepts the output of multiple 
fetcher runs for precisely this purpose.

Doug

Byron Miller wrote:
Is it possible and safe to have multiple crawlers
going?
I'm trying to figure out the most affordable and
scalable solution for a fresh index.


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers