abhinav-phi commented on PR #2128: URL: https://github.com/apache/stormcrawler/pull/2128#issuecomment-5574285812
Thanks — pushed addressing the three inline problems and the two questions: - **Proxy regression:** the loop now runs every request through the same client the first request used (hoisted as an effectively-final `fetchClient`), so dynamic proxies apply to every hop. - **Credential disclosure:** an https→http downgrade is refused outright. On any other origin change the `Authorization` and `Cookie` headers are stripped before the hop, like okhttp's own follower does on a host change. (I kept cross-origin hops themselves followable — that is okhttp's default behaviour too — but they travel without the previous origin's credentials.) - **Hop log:** the warning now only fires when the hop budget was exhausted *and* the last response is still a redirect. - **Hard-coded 5:** configurable via `http.allow.redirects.max` (default 5), shipped in crawler-default.yaml. - **`_redirTo` key collision:** the protocol now sets `ProtocolResponse.REDIRECTED_TO_KEY` (`_redirectedTo`) with a javadoc relating it to the bolts' `_redirTo` — that one names the target of a redirect which was *not* followed, this one where a followed chain ended. Downstream indexers see a key they have not seen on FETCHED pages before, which is the point, but it no longer collides. - **Filter chain per protocol instance:** yes — `URLFilters.fromConf(conf)` runs once per `configure()`, so `protocol.instances.num` instances each hold a copy. It is read-only after construction and the chain is small; noted here as considered. CI red was the Windows-format-check gap on my side; the changed files are now reformatted with google-java-format 1.35.0 (the exact version the plugin pins). A new test covers the credentials-survive-same-origin / stripped-cross-origin behaviour. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
