[
https://issues.apache.org/jira/browse/NUTCH-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369299#comment-16369299
]
Yossi Tamari commented on NUTCH-2511:
-------------------------------------
The best solution I can see for this to create a duplicate conf by using the
copy constructor, set http.content.limit to -1 (or 50MB?) and pass the new
configuration to the ProtocolFactory constructor.
Anybody has a better solution?
> SitemapProcessor limited by http.content.limit
> ----------------------------------------------
>
> Key: NUTCH-2511
> URL: https://issues.apache.org/jira/browse/NUTCH-2511
> Project: Nutch
> Issue Type: Bug
> Components: sitemap
> Affects Versions: 1.14
> Reporter: Yossi Tamari
> Priority: Major
>
> Because SitemapProcessor uses the HTTP protocol plugin, which limits the size
> of a response to http.content.limit (64KB by default), it can only handle
> sitemaps smaller than that size.
> I don't believe that is the intent of the users by setting http.content.limit
> - they want to limit document size, not sitemap size. The spec specifically
> says that sitemaps can be up to 50MB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
