GGraziadei commented on PR #2135: URL: https://github.com/apache/stormcrawler/pull/2135#issuecomment-5652317744
Rebased on main (#2128 changed the redirect handling in the same code) and addressed the points: **robots.txt timeout per path**: now the same on both. A lookup which times out on a helper thread lets the page be fetched without rules, which is what `HttpRobotRulesParser` already does with okhttp when the lookup fails. Counted by a new `robots.timeout` counter. Test `hangingRobotsLookupDoesNotFailTheUrl` checks that the fetch is attempted after the lookup timed out. **Per call vs per fetch**: with the manual redirect following from #2128 the page fetch now shares one deadline across its hops, each hop getting the time left (`redirectChainSharesTheDeadline`). The robots.txt lookup and each redirect it follows are still calls of their own with their own deadline; the yaml comment and `configuration.adoc` say so, including the worst case. **DelegatorProtocol**: the capability is `supportsFetchTimeout(url, metadata)` and the delegator resolves the delegate per URL. Since the robots.txt lookup is routed with its own metadata, the delegate on that route must support it too, otherwise the lookup would run inline on a protocol which cannot cancel it (`supportsFetchTimeoutRequiresTheRobotsDelegateToo`). **Counter**: okhttp throws a typed `FetchTimeoutException` (an `InterruptedIOException`) when the watchdog cancelled the call, and the helper path throws the same. `fetch.deadline` counts only those; `fetch.timeout` keeps counting every timeout, `http.timeout` included. `socketTimeoutIsNotTheDeadline` covers the distinction. **Clamping**: moved to `FetchTimeout.secs(conf)`, used by both `HttpProtocol` and `FetchTimeoutHelpers`, with a test on the helper side. **StuckProtocol**: `HANG_ROBOTS` is a configuration key read in `configure()`, per instance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
