The GitHub Actions job "Java CI with Maven" on stormcrawler.git/perf/fetch-timeout-call-cancel has succeeded. Run started by GitHub user GGraziadei (triggered by GGraziadei).
Head commit for run: fd7f1dd7a7b8ad8ce213dda23ec2f08a79a728c7 / Gianluca Graziadei <[email protected]> #2134 fetcher.thread.timeout: cancel the okhttp call, bounded helper pool for other protocols The bolt-level fetch timeout ran the protocol call on a single-thread executor owned by each FetcherThread and abandoned it with future.cancel(true) on expiry. okhttp does not honour the interrupt while connecting or reading, so the helper stayed blocked on the socket and the FetcherThread queued behind it at its next fetch: a host that dribbles bytes could take a thread out of service for as long as http.timeout. It also doubled the thread count of the bolt whenever the option was on, and the robots.txt lookup was not covered by the timeout at all. okhttp: fetcher.thread.timeout is applied as a per-call deadline (Call.timeout()), enforced by okio's shared watchdog: on expiry the call is cancelled, the socket closed and the fetching thread gets an InterruptedIOException at once, classified as "Socket timeout fetching". The deadline is clamped to topology.message.timeout.secs so that it can never loosen the client-level callTimeout. With http.content.partial.as.trimmed the content received before the deadline is kept and flagged as trimmed for "time", as it already was for the call timeout. Protocol gains a default supportsFetchTimeout() (false); okhttp returns true when configured, DelegatorProtocol only when every delegate does. Bolts: the timeout machinery moves to a new package-private FetchTimeoutHelpers owned by FetcherBolt and SimpleFetcherBolt. Its call() runs both the robots.txt lookup and the fetch on the calling thread when the protocol enforces the timeout itself, otherwise on a helper thread from one bounded pool per bolt (fetcher.thread.timeout.helpers, default 2 x fetcher.threads.number, 2 for SimpleFetcherBolt, threads created on demand and released after a minute idle). A deadline throws a typed TimeoutException; a full pool rejects at once with SaturatedException, reported as FETCH_ERROR "No fetch helper available". With the default protocol no helper thread is ever created. New fetchhelpers gauge and fetch.timeout / fetch.helper.rejected counters. The fetcher.thread.timeout key is defined in Constants; the public alias FetcherBolt.FETCH_TIMEOUT_PARAM_KEY (#1861) is removed. Documentation for the parameter, which was missing, is added to configuration.adoc. Tests: HttpProtocolFetchTimeoutTest (cancellation at the deadline, clamp to the message timeout, partial content), FetchTimeoutHelpersTest (every branch of call()), fetcher bolt tests with a protocol that hangs and ignores interruption (stuck fetch not blocking the following ones, bounded pool with rejection, hanging robots.txt reported at the timeout), slow robots.txt bounded with okhttp, no helper threads with okhttp, delegator capability. Fixes #2134. Report URL: https://github.com/apache/stormcrawler/actions/runs/34062685893 With regards, GitHub Actions via GitBox
