jnioche opened a new pull request, #1987: URL: https://github.com/apache/stormcrawler/pull/1987
# Make the robots.txt parser implementation configurable Currently `AbstractHttpProtocol` hardcodes `HttpRobotRulesParser` as the class used to fetch and parse robots.txt for every HTTP-based Protocol implementation. This makes it impossible to customize robots.txt handling (e.g. to support directives that crawler-commons doesn't recognise, such as the emerging [Content-Signal ](https://contentsignals.org/) proposal) without forking core classes. This PR makes the implementation pluggable via configuration, following the same pattern already used for `http.protocol.implementation` and `http.proxy.manager`: - Added a new config key, `http.robots.parser.class`, defaulting to `org.apache.stormcrawler.protocol.HttpRobotRulesParser `(unchanged default behaviour). - `AbstractHttpProtocol` now instantiates the configured class via `InitialisationUtil.initializeFromQualifiedName(...),` typed against the abstract `RobotRulesParser` rather than the concrete `HttpRobotRulesParser`. - `HttpRobotRulesParser's` no-arg constructor is now public (required for reflective instantiation). - Added `RobotRules.getWrapped()`, returning the wrapped `BaseRobotRules. HttpRobotRulesParser.getRobotRulesSet() `always wraps whatever `parseRules() `returns in a plain `RobotRules` for content-length/cache-hit tracking; without this accessor, a custom `RobotRulesParser` that returns its own `BaseRobotRules` subclass (to carry extra parsed data) would have that subtype hidden by the outer wrapper with no way to retrieve it. - Documented the new setting in `crawler-default.yaml`. No behavioural change with the default configuration; existing tests pass unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
