Hi folks, I opened PR#2135 [1], which fixes #2134.
The bolt-level fetch timeout introduced in #1861 did not actually free the fetcher thread: okhttp ignores the interrupt sent by future.cancel(true), so the helper stayed blocked on the socket until http.timeout and the next fetch queued behind it. It also doubled the thread count of the bolt whenever the option was on, and the robots.txt lookup was not covered at all. The PR changes this in two ways: - With okhttp, fetcher.thread.timeout is applied as a per-call deadline via Call.timeout(). At expiry the call is cancelled and the socket closed, and the fetching thread gets an InterruptedIOException immediately. The deadline is clamped to topology.message.timeout.secs, and partial content is kept as trimmed when http.content.partial.as.trimmed is on. - Protocols that cannot cancel their own I/O run on a bounded helper pool (fetcher.thread.timeout.helpers, default 2 x fetcher.threads.number). Threads are created on demand and released after a minute idle; a full pool rejects at once instead of queueing. With the default protocol no helper thread is ever created. Protocol gains a default supportsFetchTimeout() to pick the path, the robots.txt lookup is now under the same deadline, and there are new fetch.timeout / fetch.helper.rejected counters and a fetchhelpers gauge. The parameter is documented in configuration.adoc, which it was not before. A short design note with diagrams is attached [2]: the problem, the decision flow, both paths, configuration, and the tests that cover them. Reviews welcome. [1] https://github.com/apache/stormcrawler/pull/2135 [2] https://docs.google.com/document/d/1XXwdz8-FQaRurvSpljVUdE3IuRmj-lG6/edit?usp=sharing&ouid=114888462700939773114&rtpof=true&sd=true Cheers, Gianluca
