Thank you, Sebastian, that sounds like a great suggestion! You're right that 3000 is a small segment size. I am using 3000 per slave just in this still-early testing phase. I don't know the actual size of my linkdb, but my crawldb has over 48 million urls so far, of which over 1.5 million have been fetched.
I think I need the linkdb because incoming anchors are important for search-engine relevance, right? ________________________________ From: Sebastian Nagel <[email protected]> Hi Michael, what is the size of your linkdb? If it's large (significantly larger than the segment) the reason is easily explained: the linkdb needs to be rewritten on every invertlinks step. That's an expensive action becoming more expensive for larger crawls. Unless you really need the linkdb to add anchor texts to your index you could: - either limit the linkdb size by excluding internal links - or update it less frequently (multiple segments in one turn) A segment size of 3000 URLs seems small for a distributed crawl with a large number of different hosts or domains. You may observe similar problems updating the CrawlDb, although later because the CrawlDb is usually smaller, esp. if the linkdb includes also internal links. Best, Sebastian On 04/03/2017 02:08 AM, Michael Coffey wrote: > In my situation, I find that linkdb merge takes much more time than fetch and > parse combined, even though fetch is fully polite. > > What is the standard advice for making linkdb-merge go faster? > > I call invertlinks like this: > __bin_nutch invertlinks "$CRAWL_PATH"/linkdb "$CRAWL_PATH"/segments/$SEGMENT > > invertlinks seems to call mergelinkdb automatically. > > I currently have about 3-6 slaves for fetching, though that will increase > soon. I am currently using small segment sizes (3000 urls) but I can increase that if it would help. > > I have the following properties that may be relevant. > > <property> > <name>db.max.outlinks.per.page</name> > <value>1000</value> > </property> > > <property> > <name>db.ignore.external.links</name> > <value>false</value> > </property> > > > The following props are left as default in nutch-default.xml > > <property> > <name>db.update.max.inlinks</name> > <value>10000</value> > </property> > > <property> > <name>db.ignore.internal.links</name> > <value>false</value> > </description> > </property> > > <property> > <name>db.ignore.external.links</name> > <value>false</value> > </description> > </property> >
