Markus Ackermann created CXF-9268:
-------------------------------------

             Summary: AsyncHTTPConduit (hc5) deadlocks in SharedInputBuffer 
with httpcore5 5.4.4 (32 KiB HTTP/1.1 buffer); receive timeout does not fire
                 Key: CXF-9268
                 URL: https://issues.apache.org/jira/browse/CXF-9268
             Project: CXF
          Issue Type: Bug
          Components: Transports
    Affects Versions: 4.2.4
         Environment: Java 21, cxf-rt-transports-http-hc5 4.2.4, httpcore5 
5.4.4, httpclient5 5.6.5, AsyncHTTPConduit (USE_ASYNC=true), HTTP/1.1, no 
chunkLength set in HTTPClientPolicy
            Reporter: Markus Ackermann
         Attachments: AsyncHTTPConduitLargeResponseHangTest.java, 
threaddump-httpcore5-5.4.4.txt

h3. Problem

After upgrading httpcore5 from 5.4.3 to 5.4.4 (pulled in by Spring Boot 4.1.2), 
SOAP calls via the hc5 {{AsyncHTTPConduit}} hang forever as soon as responses 
are larger than about 20 KB. The configured {{ReceiveTimeout}} does not fire. 
In production, all hc5 I/O dispatcher threads ended up blocked within minutes 
under load. After that, no request on any connection completed until the JVM 
was restarted. We saw calls hang for almost 2 hours.

httpcore5 5.4.4 increased the default HTTP/1.1 buffer size from 8 KiB to 32 KiB 
("Increased the default HTTP/1.1 buffer size to 32 KiB", 
apache/httpcomponents-core#687). The I/O reactor therefore now passes data 
blocks of up to 32 KiB to {{CXFHttpAsyncResponseConsumer.consume()}}. That 
exposes a lost-wakeup in {{SharedInputBuffer}}.

h3. Analysis

Reader thread ({{SharedInputBuffer.read(byte[], int, int)}}, line 244): if the 
buffer is empty, it wraps the caller's array as {{waitingBuffer}} and calls 
{{waitForData()}}. That method signals {{suspendInput}} and then waits on 
{{condition}} without a timeout (line 170).

I/O dispatcher thread ({{SharedInputBuffer.consumeContent()}}):
# copies as much as fits into {{waitingBuffer}} (lines 84-87),
# copies the *rest* of {{src}} into the internal buffer. {{ensureCapacity()}} 
grows that buffer to exactly the remaining size, so afterwards the buffer has 
no free space left (line 91),
# because of {{!buffer().hasRemaining() && !endOfStream}}, it then calls 
{{suspendInput.await()}} (line 101),
# and only *after* that wait does it call {{condition.signalAll()}} (line 106).

The reader is waiting on {{condition}} and would see that {{waitingBuffer}} has 
been filled, but it is never woken up. The I/O thread waits on 
{{suspendInput}}, which only the reader would signal. Both threads wait forever.

This deadlock occurs whenever a single {{src}} block is at least 
{{waitingBuffer.remaining() + chunkLength}}. With the default {{chunkLength}} 
of 16320 and the usual read buffers of a few KiB (for example 4096 bytes in 
{{IOUtils.copyAtLeast()}} via {{WireTapIn}}, or Woodstox's input buffer), that 
threshold is about 20 KiB:
* With 8 KiB reactor blocks (httpcore5 <= 5.4.3), the threshold was never 
reached.
* With 32 KiB blocks (httpcore5 5.4.4), it is reached as soon as a response is 
large enough.

The receive timeout cannot fire either: it is enforced by the I/O reactor, and 
the reader in {{AsyncWrappedOutputStream.getHttpResponse()}} uses 
{{Object.wait()}} without a timeout. Since the I/O dispatcher thread is shared 
by all connections assigned to it, one stuck exchange blocks every other 
connection on that dispatcher.

h3. Thread dump (excerpt)

{code}
// taken ~10 s into a hanging run of the attached test 
(cxf-rt-transports-http-hc5 4.2.4, httpcore5 5.4.4, httpclient5 5.6.5, JDK 21)
// I/O dispatch thread blocked in SharedInputBuffer.consumeContent
"httpclient-dispatch-2" daemon
   java.lang.Thread.State: WAITING (parking)
    at 
org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.consumeContent(SharedInputBuffer.java:101)
    at 
org.apache.cxf.transport.http.asyncclient.hc5.CXFHttpAsyncResponseConsumer.consume(CXFHttpAsyncResponseConsumer.java:81)
    at 
org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.consume(HttpAsyncMainClientExec.java:275)
    at 
org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.consumeData(ClientHttp1StreamHandler.java:270)
    at 
org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeData(ClientHttp1StreamDuplexer.java:364)
    at 
org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:333)
    at 
org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
    at 
org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
    at 
org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:139)
    at 
org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
    ...

// Calling thread blocked in SharedInputBuffer.waitForData
"test-caller" daemon
   java.lang.Thread.State: WAITING (parking)
    at 
org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.waitForData(SharedInputBuffer.java:170)
    at 
org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.read(SharedInputBuffer.java:244)
    at 
org.apache.cxf.transport.http.asyncclient.hc5.AsyncHTTPConduit$AsyncWrappedOutputStream$6.read(AsyncHTTPConduit.java:734)
    at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155)
    at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:385)
    at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:116)
    at com.ctc.wstx.io.MergedReader.read(MergedReader.java:104)
    at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:88)
    at 
com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
    ...
{code}

In production we saw exactly matching pairs: 8 dispatcher threads at 
{{SharedInputBuffer.java:101}} and 8 reader threads at 
{{SharedInputBuffer.java:170}} (via {{read(byte[],int,int)}} at line 244).

h3. Reproduction

The attached {{AsyncHTTPConduitLargeResponseHangTest.java}} is a self-contained 
JUnit 5 test. It needs only CXF jaxws frontend, the http-hc5 transport, the JDK 
{{HttpServer}} and JUnit 5. It starts a local HTTP server that returns a 300 KB 
SOAP response with {{Content-Length}} and calls it from 20 threads × 5 times 
through {{JaxWsProxyFactoryBean}} with {{USE_ASYNC=true}}.

|| httpcore5 / httpclient5 || chunkLength not set (16320) || chunkLength = 
65536 ||
| 5.4.3 / 5.6.4 | passes | passes |
| 5.4.4 / 5.6.5 | *hangs* (all 20 caller threads blocked, test times out after 
15 s; reproducible) | passes |

h3. Workaround

Set {{HTTPClientPolicy.setChunkLength()}} well above the httpcore5 HTTP/1.1 
buffer size, e.g. 65536. Since then, the problem has not occurred in our 
production system.

h3. Suggested fix

In {{SharedInputBuffer.consumeContent()}}, signal {{condition}} before blocking 
on {{suspendInput}}:

{code:java}
if (!buffer().hasRemaining() && !this.endOfStream) {
    this.condition.signalAll();   // wake up a reader waiting on waitingBuffer
    try {
        suspendInput.await();
    } catch (InterruptedException ex) {
        throw new IOException("Interrupted while waiting buffer to be drained 
");
    }
}
this.condition.signalAll();
{code}

This fix is untested. Independently of that, it may be worth not blocking the 
I/O dispatcher thread at all and using {{CapacityChannel}} for flow control 
instead ({{updateCapacity()}} currently always grants {{Integer.MAX_VALUE}}).

Attachments:
- AsyncHTTPConduitLargeResponseHangTest.java
- threaddump-httpcore5-5.4.4.txt (full dump from the reproduction)



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to