abhinav-phi opened a new pull request, #2121: URL: https://github.com/apache/stormcrawler/pull/2121
Fixes #2087. `HttpRobotRulesParser.setConf` put `http.robots.content.limit` (-1 by default) into the fetch metadata as `http.content.limit`, and `HttpProtocol.getProtocolOutput` applies that value unconditionally — so every robots.txt fetch ran with no limit at all, even when the operator configured a finite `http.content.limit`, although the shipped comment said the default was 'same as http.content.limit'. With no limit, the decompressed body is buffered until `Constants.MAX_ARRAY_SIZE`, just under 2 GB. - `http.robots.content.limit: -1` now means 'inherit the global limit'; the key is only written into the fetch metadata when a robots-specific limit is configured - the shipped default of `http.robots.content.limit` is 524288 (512 kiB), the minimum the robots.txt RFC draft asks crawlers to fetch and parse - `HttpProtocol` ignores a metadata `http.content.limit` of -1 when the global limit is finite, so per-URL metadata can tighten the limit but not remove it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
