[ https://issues.apache.org/jira/browse/NUTCH-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814388#comment-16814388 ]
Hudson commented on NUTCH-2666: ------------------------------- FAILURE: Integrated in Jenkins build Nutch-trunk #3619 (See [https://builds.apache.org/job/Nutch-trunk/3619/]) NUTCH-2666 Increase default value for http.content.limit / (snagel: [https://github.com/apache/nutch/commit/13a9a6daf2ca2f764d052ee338b51dc9f91824d5]) * (edit) src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/Ftp.java * (edit) src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java * (edit) conf/nutch-default.xml * (edit) src/plugin/protocol-file/src/java/org/apache/nutch/protocol/file/File.java > Increase default value for http.content.limit / ftp.content.limit / > file.content.limit > -------------------------------------------------------------------------------------- > > Key: NUTCH-2666 > URL: https://issues.apache.org/jira/browse/NUTCH-2666 > Project: Nutch > Issue Type: Improvement > Components: fetcher > Affects Versions: 1.15 > Reporter: Marco Ebbinghaus > Assignee: Sebastian Nagel > Priority: Minor > Fix For: 1.16 > > > The default value for http.content.limit in nutch-default.xml (The length > limit for downloaded content using the http:// > protocol, in bytes. If this value is nonnegative (>=0), content longer > than it will be truncated; otherwise, no truncation at all. Do not > confuse this setting with the file.content.limit setting.) is set to 64kb. > Maybe this default value should be increased as many pages today are greater > than 64kb. > This fact hit me when trying to crawl a single website whose pages are much > greater than 64kb and because of that with every crawl cycle the count of > db_unfetched urls decreased until it hit zero and the crawler became inactive > (because the first 64 kB contained always the same set of navigation links) > The description might also be updated as this is not only the case for the > http protocol, but also for https. -- This message was sent by Atlassian JIRA (v7.6.3#76005)