abhinav-phi opened a new pull request, #2127: URL: https://github.com/apache/stormcrawler/pull/2127
Fixes #2080. The connection-time IP filter was the only check that looked at the address the fetcher actually connects to, and both of its keys shipped commented out in `crawler-default.yaml`, so at defaults the interceptor was never installed. The regex exclusion list in the archetypes matched literal `localhost` and four dotted-quad ranges and nothing else — link-local, CGNAT, IPv6 unique-local and the abbreviated and integer IPv4 forms the JVM resolver maps to 127.0.0.1 all passed it, and `InetAddress.getByName` maps `127.1` and `2130706433` to loopback without any DNS control. - `crawler-default.yaml` ships `http.filter.ipaddress.exclude` enabled: `localhost,sitelocal,linklocal,100.64.0.0/10,fd00::/8`. A fetched page decides which hosts the fetcher connects to, and these ranges host unauthenticated services (e.g. cloud instance metadata at 169.254.169.254, reachable whatever DNS name a link used, since the check runs on the resolved socket address) which a public index must not leak into - crawling an intranet or a loopback service needs an explicit opt-out — the comment next to the key documents how - the archetype `default-regex-filters.txt` (and its opensearch/solr copies) gains rules for 169.254.0.0/16, 100.64.0.0/10, 0.0.0.0/8, IPv6 `fc00::/7` and `fe80::/10`, abbreviated loopback (`127.1`) and integer IPv4 hosts (`2130706433`), and states that a regex list judges the bytes of the URL only — it can never enforce where a host name resolves, so the IP filter stays the authoritative check **Release note needed:** enabling the exclude list by default changes behaviour for anyone crawling an intranet or a loopback service; the way to opt out is documented next to the key. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
