You would have to check the generator code to make sure. But why would you want to distribute the queue for a single domain to multiple mappers? A single local running mapper without parsing on a low-end machine can easily fetch 20-40 records per second from the same domain (if it allows you to do it). At that speed you can easily fetch a few million records in a day orso.
-----Original message----- > From:shri_s_ram <[email protected]> > Sent: Thu 18-Oct-2012 23:11 > To: [email protected] > Subject: RE: Nutch generate fetch lists for a single domain (but with > multiple urls) crawl > > Thanks.. But I thought there would be a way around it.. > Is it possible even to have multiple fetch lists generated (for this > problem) at all by tweaking some parameters? > > [I am thinking of something like partition.url.mode - byRandom] > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Nutch-generate-fetch-lists-for-a-single-domain-but-with-multiple-urls-crawl-tp4014573p4014582.html > Sent from the Nutch - User mailing list archive at Nabble.com. >
