Yossi Tamari commented on NUTCH-2511:

Opened a PR for this: [https://github.com/apache/nutch/pull/479].

I created a new conf parameter, sitemap.size.max, but did not add it to 
nutch-default.xml, since I already gave it a default of 
SiteMapParser.MAX_BYTES_ALLOWED and I doubt many people will want to change 
this. However, I can add it to nutch-default.xml if you think it's best.

> SitemapProcessor limited by http.content.limit
> ----------------------------------------------
>                 Key: NUTCH-2511
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2511
>             Project: Nutch
>          Issue Type: Bug
>          Components: sitemap
>    Affects Versions: 1.14
>            Reporter: Yossi Tamari
>            Priority: Major
>             Fix For: 1.17
> Because SitemapProcessor uses the HTTP protocol plugin, which limits the size 
> of a response to http.content.limit (64KB by default), it can only handle 
> sitemaps smaller than that size. 
> I don't believe that is the intent of the users by setting http.content.limit 
> - they want to limit document size, not sitemap size. The spec specifically 
> says that sitemaps can be up to 50MB.

This message was sent by Atlassian Jira

Reply via email to