dpol1 commented on code in PR #2127: URL: https://github.com/apache/stormcrawler/pull/2127#discussion_r3976680622
########## core/src/main/resources/crawler-default.yaml: ########## @@ -165,8 +165,14 @@ config: # topology from starting. # Only addresses matching an include rule are fetched (empty means all are # allowed), addresses matching an exclude rule are always blocked. - # http.filter.ipaddress.include: - # http.filter.ipaddress.exclude: "localhost,sitelocal,linklocal,anylocal,multicast,100.64.0.0/10,fc00::/7" + # The exclude list is enabled by default: a fetched page decides which hosts + # the fetcher connects to, and loopback, private and link-local ranges host + # unauthenticated services (e.g. cloud instance metadata) which a public + # index must not leak into. To crawl an intranet, override the exclude list + # explicitly (exclude rules win over include rules, so setting only + # http.filter.ipaddress.include does not undo this default), e.g. + # http.filter.ipaddress.exclude: "" + http.filter.ipaddress.exclude: "localhost,sitelocal,linklocal,anylocal,multicast,100.64.0.0/10,0.0.0.0/8,fc00::/7,::/128" Review Comment: with a proxy configured (`http.proxy.host`) the filter sees the proxy's socket address, not the target's, so an HTTP proxy on a private range now fails every fetch with this default. either skip the check when `connection.route().proxy()` is set and document that proxied fetches are unfiltered, or state that the list must be cleared behind a proxy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
