You would create multiple segments by passing -numfetchers as part of the generate segments commmand.
I dump my fetchlists onto a Coda fs (distributed) file system that is mounted by the crawling servers and it works great.. http://www.nutch.org/cgi-bin/twiki/view/Main/GenerateOptions -byron --- Jose Vega <[EMAIL PROTECTED]> wrote: > Hello... > > What is the right way to implement multiple > crawlers (in diferent > hosts), and avoid to download common urls (pages) > once and another time? > ------------------------------------------------------- This SF.Net email is sponsored by OSTG. Have you noticed the changes on Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now, one more big change to announce. We are now OSTG- Open Source Technology Group. Come see the changes on the new OSTG site. www.ostg.com _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
