dpol1 opened a new issue, #2305: URL: https://github.com/apache/stormcrawler/issues/2305
The archetypes set `sitemap.discovery: true` and `http.content.limit: 65536`, and the limit applies to sitemaps too. An XML sitemap above 64 KiB is cut and fails to parse, so none of its URLs are discovered. `SiteMapParserBolt` marks the fetch as FETCH_ERROR and removes `isSitemap`, so the retry treats it as a page. sitemaps.org allows up to 50 MB. In a 12-minute crawl of GQ32K hosts, there were 1,457 sitemap parsing failures, all on responses cut at the limit. rzo1 got 4,183 such errors in #2268. The 3.7.0 archetypes have the same settings, and there the fetch ends as ERROR. This affects the configuration shipped with the archetypes. `crawler-default.yaml` leaves the content size unlimited. A higher limit in the archetypes would also apply to pages, so I suggest a separate limit for sitemaps, as we have with `http.robots.content.limit` and as Nutch has with `sitemap.size.max`. The robots.txt limit goes through the per-URL `http.content.limit` metadata. Sitemaps found in robots.txt or in a sitemap index already carry `isSitemap=true` before their fetch, so a sitemap limit could work the same way. A redirect target doesn't carry it, and neither does the retry after a parsing failure, so the limit would have to cover those too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
