-----Original message----- > From:pepe3059 <[email protected]> > Sent: Wed 06-Jun-2012 02:58 > To: [email protected] > Subject: RE: threads disminution when fetching page > > me again :) > > at the end of fetch process, is the regex-urlfilter considered?
No. At the end of the fetch the mapper output is writting to intermediate files, they get merged and sorted and are sent to the reducer which consumes all output and then finally writes it to disk. > because i have 16+ regular expressions but update process is not reached yet. > is > considered in updatedb? > another datum; disk usage from the last segment: 21Gb You should make your segments smaller by generating less records, consider the -topn parameter for the generator. > are 20 threads heavy load for a six core xeon? (i assign the rule "-threads > 20") No, not too heavy, it can be much higher. > > > > > sorry for my bad english :P and thank you for your answers > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/threads-disminution-when-fetching-page-tp3987381p3987887.html > Sent from the Nutch - User mailing list archive at Nabble.com. >
