You need also to set <property> <name>db.max.outlinks.per.page</name> <value>2000</value> <description>The maximum number of outlinks that we'll process for a page. </description> </property>
-----Original Message----- From: "Håvard W. Kongsgård" [mailto:[EMAIL PROTECTED] Sent: Tuesday, November 22, 2005 11:01 AM To: [email protected] Subject: Re: Intranet craw folder Do you mean http.content.limit? I have set it to -1 already. There are no "Content truncated at 65536 bytes. Parser can't handle incomplete" errors in the log. Stefan Groschupf wrote: > Check the maximal content limit in nutch-default.xml > > Am 22.11.2005 um 16:38 schrieb Håvard W. Kongsgård: > >> Hi, I am still testing nutch 0.7.1 but now I have another problem. >> When I do a normal intranet crawl on some web folders with 2000 >> pdfs, nutch only fetches 47 pdfs from each folder. >> >> > > > >
