Hudson commented on NUTCH-2666:

FAILURE: Integrated in Jenkins build Nutch-trunk #3619 (See 
NUTCH-2666 Increase default value for http.content.limit / (snagel: 
* (edit) src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/Ftp.java
* (edit) 
* (edit) conf/nutch-default.xml
* (edit) 

> Increase default value for http.content.limit / ftp.content.limit / 
> file.content.limit
> --------------------------------------------------------------------------------------
>                 Key: NUTCH-2666
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2666
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 1.15
>            Reporter: Marco Ebbinghaus
>            Assignee: Sebastian Nagel
>            Priority: Minor
>             Fix For: 1.16
> The default value for http.content.limit in nutch-default.xml (The length 
> limit for downloaded content using the http://
>  protocol, in bytes. If this value is nonnegative (>=0), content longer
>  than it will be truncated; otherwise, no truncation at all. Do not
>  confuse this setting with the file.content.limit setting.) is set to 64kb. 
> Maybe this default value should be increased as many pages today are greater 
> than 64kb.
> This fact hit me when trying to crawl a single website whose pages are much 
> greater than 64kb and because of that with every crawl cycle the count of 
> db_unfetched urls decreased until it hit zero and the crawler became inactive 
> (because the first 64 kB contained always the same set of navigation links)
> The description might also be updated as this is not only the case for the 
> http protocol, but also for https.

This message was sent by Atlassian JIRA

Reply via email to