rzo1 commented on code in PR #2127: URL: https://github.com/apache/stormcrawler/pull/2127#discussion_r3976885444
########## core/src/main/resources/crawler-default.yaml: ########## @@ -165,8 +165,14 @@ config: # topology from starting. # Only addresses matching an include rule are fetched (empty means all are # allowed), addresses matching an exclude rule are always blocked. - # http.filter.ipaddress.include: - # http.filter.ipaddress.exclude: "localhost,sitelocal,linklocal,anylocal,multicast,100.64.0.0/10,fc00::/7" + # The exclude list is enabled by default: a fetched page decides which hosts + # the fetcher connects to, and loopback, private and link-local ranges host + # unauthenticated services (e.g. cloud instance metadata) which a public + # index must not leak into. To crawl an intranet, override the exclude list + # explicitly (exclude rules win over include rules, so setting only + # http.filter.ipaddress.include does not undo this default), e.g. + # http.filter.ipaddress.exclude: "" + http.filter.ipaddress.exclude: "localhost,sitelocal,linklocal,anylocal,multicast,100.64.0.0/10,0.0.0.0/8,fc00::/7,::/128" Review Comment: Fixed in ed35b139, taking the first option. The interceptor now lets a connection through when `route().proxy()` is not `DIRECT`: the proxy resolves the target itself, so the only address the filter ever saw was the proxy's. A log line at configure time says that the filter does not apply to proxied fetches, and the yaml and configuration.adoc document that the proxy's own egress rules decide there. Pinned by `fetchThroughProxyOnLoopbackIsNotFiltered`, which fetches through a proxy on 127.0.0.1 under the shipped exclude list and fails on the previous code with "Forbidden connection to IP address 127.0.0.1". The same commit updates configuration.adoc for your other point: the table shows the new default of `http.filter.ipaddress.exclude`, and the IP Address Filtering section no longer calls the filter optional and explains how to override it for an intranet crawl. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
