abhinav-phi opened a new pull request, #2124: URL: https://github.com/apache/stormcrawler/pull/2124
Fixes #2081. `crawler-default.yaml` shipped `protocols: "http,https,file"`, and `FileProtocol` read whatever path the worker user could read: a fetched page can put a `file://` URL into the frontier (absolute `file://` hrefs resolve to themselves in the parser), and the response body — including the topology configuration and the backend credentials in it — went on to be parsed and indexed. The only gate was URL filtering, and the library default ships no URL filters at all. - the shipped `protocols` list is now `http,https`; enabling the file scheme is a deliberate act - `FileProtocol` gains `file.protocol.root`: when set, only paths canonicalising below that directory are served (symlinks included), everything else returns 403 - when no root is set, the file scheme serves nothing — reading any path requires having chosen a root on purpose **Release note needed:** topologies crawling local corpora on the old default stop working until they add `file` to `protocols` and set `file.protocol.root`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
