abhinav-phi opened a new pull request, #2128: URL: https://github.com/apache/stormcrawler/pull/2128
Fixes #2088. With `http.allow.redirects` enabled, okhttp followed the whole redirect chain inside the client and the crawler only ever saw the final response, returned under the original URL. No intermediate target went through `URLFilters`, so the scheme exclusions, host confinement and depth rules configured for the crawl did not apply to the hops a fetched page steered the fetcher to — an operator who turned the setting on to keep redirect chains out of the status store also turned off URL policy for those hops, two unrelated concerns. Redirects are now followed manually in `getProtocolOutput` (the client's internal following is disabled): - each target is resolved against the URL it was reached from and run through the URL filter chain; only accepted targets are fetched - a rejected target ends the chain: the redirect response itself is returned, which the caller handles exactly like it does when immediate follows are off (emits the target to the status stream, where the full policy applies) - the chain is capped at 5 hops (a chain which does not end within that returns its last redirect response) - the final URL is recorded in the response metadata under `_redirTo`, so callers can tell that the content is not from the URL they asked for; the cookie origin is scoped per hop to the URL that actually answered The `IPFilterRules` interceptor already ran per hop as a network interceptor; it keeps doing so. Docs updated (configuration.adoc, internals.adoc) — the previous wording said immediate follows bypass URL filtering, which is no longer true for OkHttp. **Note:** chains longer than 5 hops stop being followed in-client; their last redirect is handed to the status store, so no URL is lost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
