rzo1 opened a new pull request, #2169: URL: https://github.com/apache/stormcrawler/pull/2169
The fetcher stores response headers prefixed with `protocol.md.prefix`, so reading `Content-Type` without it never found a value in a shipped topology. The declared type was ignored and every sitemap was identified by guessing. It also left the media type check in the content sniffing dead: a page served as `text/html` carrying the sitemaps namespace was promoted to a sitemap. Reads the key with the configured prefix, as `JSoupParserBolt` does. The other half of #2100 is not addressed here. `processGzippedXML` streams into SAX, so there is no decompressed buffer to bound — what grows is the accumulated `SiteMapURL` set, and neither `MAX_BYTES_ALLOWED` nor `MAX_URLS` is applied on the XML path. Raised upstream as crawler-commons/crawler-commons#611, with #2166 for a cap on our side in the meantime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
