GGraziadei opened a new issue, #2134: URL: https://github.com/apache/stormcrawler/issues/2134
`fetcher.thread.timeout` (#996, #1861) wraps `protocol.getProtocolOutput()` in a `Future` run on a single-thread executor owned by each `FetcherThread`, and cancels it with `future.cancel(true)` when the deadline passes. Two problems with that: - `cancel(true)` is a `Thread.interrupt()`, which okhttp does not honour while connecting or reading. The helper thread stays blocked on the socket until okhttp's own timeouts fire, and the `FetcherThread` queues behind it at its next fetch because the executor is single-threaded. One host that dribbles bytes can take a `FetcherThread` out of service for as long as `http.timeout`, or `topology.message.timeout.secs` (300s by default) which is the value the okhttp `callTimeout` is currently set from. - It doubles the thread count of the bolt whenever the option is on: 50 `FetcherThread`s plus 50 `FetcherTimeout` helpers. okhttp exposes the right primitive: `Call.timeout()` sets a per-call deadline enforced by okio's single shared watchdog thread, and on expiry the call is cancelled, the socket closed, and the fetching thread gets an `InterruptedIOException` immediately. Proposal: `okhttp.HttpProtocol` applies `fetcher.thread.timeout` as a per-call deadline, and `Protocol` gains a `default boolean supportsFetchTimeout()` (false by default, true for okhttp when configured). `FetcherBolt` and `SimpleFetcherBolt` keep the `Future` path only for protocols that do not support it, creating the helper lazily. For the default okhttp protocol the fetch runs in the `FetcherThread` itself, the timeout is a real cancellation, and no helper threads exist. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
