William Montaz created CXF-7122:
-----------------------------------
Summary: Infinite loop due to AsyncHTTPConduit read timeout with
exhausted connection pool
Key: CXF-7122
URL: https://issues.apache.org/jira/browse/CXF-7122
Project: CXF
Issue Type: Bug
Components: Transports
Reporter: William Montaz
Priority: Critical
Using AsyncHTTPConduit, when the underlying connection pool gets exhausted,
requests waiting for a connection will lead to an infinite loop if they reach
receive timeout.
The problem occured on all versions of CXF above 3.0.5 (we did not tested other
ones).
Let's imagine a backend that's broken and leads to timeout for all requests.
When handling requests, the cxf worker thread will eventually go in wait state
(AsyncHTTPConduit:618), with a timeout that matches the
HTTPClientPolicy.setReceiveTimeout() value, waiting for the NIO stack to
complete and call notifyAll via responseCallback (AsyncHTTPConduit:455).
The timeout on the wait is the big problem :
With our broken backend, the connection pool is exhausted waiting for other
requests to timeout. When a new request is made by cxf against this backend,
after timeout time this will happen :
- on the one side the reactor threads will get a connection from the pool and
try to write to the output stream. Waiting in the pool is not considered as
receive timeout.
- on the other side the cxf worker thread will wake up (because of the
timedout wait), and shutdown SharedOutputBuffer and SharedInputBuffer
(AsyncHTTPClient:624)
- reactor threads will go to infinite loop because they will try to
produceContent from a shutdown buffer (SharedOutputBuffer:120)
From there, application recovery is compromised.
To fix that, timeout should be handled only via the client callback
(AsyncHTTPConduit:463).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
