[ 
https://issues.apache.org/jira/browse/CXF-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137450#comment-18137450
 ] 

Andriy Redko commented on CXF-9268:
-----------------------------------

Yes, I believe the problem is coming from HTTP/1.1 default buffer changes 
introduced in HttpCore5. The conduit sets it to 32Kb by default, see please 
https://github.com/apache/cxf/pull/3508

> AsyncHTTPConduit (hc5) deadlocks in SharedInputBuffer with httpcore5 5.4.4 
> (32 KiB HTTP/1.1 buffer); receive timeout does not fire
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CXF-9268
>                 URL: https://issues.apache.org/jira/browse/CXF-9268
>             Project: CXF
>          Issue Type: Bug
>          Components: Transports
>    Affects Versions: 4.2.4
>         Environment: Java 21, cxf-rt-transports-http-hc5 4.2.4, httpcore5 
> 5.4.4, httpclient5 5.6.5, AsyncHTTPConduit (USE_ASYNC=true), HTTP/1.1, no 
> chunkLength set in HTTPClientPolicy
>            Reporter: Markus Ackermann
>            Assignee: Freeman Yue Fang
>            Priority: Major
>         Attachments: AsyncHTTPConduitLargeResponseHangTest.java, 
> threaddump-httpcore5-5.4.4.txt
>
>
> h3. Problem
> After upgrading httpcore5 from 5.4.3 to 5.4.4 (pulled in by Spring Boot 
> 4.1.2), SOAP calls via the hc5 {{AsyncHTTPConduit}} hang forever as soon as 
> responses are larger than about 20 KB. The configured {{ReceiveTimeout}} does 
> not fire. In production, all hc5 I/O dispatcher threads ended up blocked 
> within minutes under load. After that, no request on any connection completed 
> until the JVM was restarted. We saw calls hang for almost 2 hours.
> httpcore5 5.4.4 increased the default HTTP/1.1 buffer size from 8 KiB to 32 
> KiB ("Increased the default HTTP/1.1 buffer size to 32 KiB", 
> apache/httpcomponents-core#687). The I/O reactor therefore now passes data 
> blocks of up to 32 KiB to {{CXFHttpAsyncResponseConsumer.consume()}}. That 
> exposes a lost-wakeup in {{SharedInputBuffer}}.
> h3. Analysis
> Reader thread ({{SharedInputBuffer.read(byte[], int, int)}}, line 244): if 
> the buffer is empty, it wraps the caller's array as {{waitingBuffer}} and 
> calls {{waitForData()}}. That method signals {{suspendInput}} and then waits 
> on {{condition}} without a timeout (line 170).
> I/O dispatcher thread ({{SharedInputBuffer.consumeContent()}}):
> # copies as much as fits into {{waitingBuffer}} (lines 84-87),
> # copies the *rest* of {{src}} into the internal buffer. {{ensureCapacity()}} 
> grows that buffer to exactly the remaining size, so afterwards the buffer has 
> no free space left (line 91),
> # because of {{!buffer().hasRemaining() && !endOfStream}}, it then calls 
> {{suspendInput.await()}} (line 101),
> # and only *after* that wait does it call {{condition.signalAll()}} (line 
> 106).
> The reader is waiting on {{condition}} and would see that {{waitingBuffer}} 
> has been filled, but it is never woken up. The I/O thread waits on 
> {{suspendInput}}, which only the reader would signal. Both threads wait 
> forever.
> This deadlock occurs whenever a single {{src}} block is at least 
> {{waitingBuffer.remaining() + chunkLength}}. With the default {{chunkLength}} 
> of 16320 and the usual read buffers of a few KiB (for example 4096 bytes in 
> {{IOUtils.copyAtLeast()}} via {{WireTapIn}}, or Woodstox's input buffer), 
> that threshold is about 20 KiB:
> * With 8 KiB reactor blocks (httpcore5 <= 5.4.3), the threshold was never 
> reached.
> * With 32 KiB blocks (httpcore5 5.4.4), it is reached as soon as a response 
> is large enough.
> The receive timeout cannot fire either: it is enforced by the I/O reactor, 
> and the reader in {{AsyncWrappedOutputStream.getHttpResponse()}} uses 
> {{Object.wait()}} without a timeout. Since the I/O dispatcher thread is 
> shared by all connections assigned to it, one stuck exchange blocks every 
> other connection on that dispatcher.
> h3. Thread dump (excerpt)
> {code}
> // taken ~10 s into a hanging run of the attached test 
> (cxf-rt-transports-http-hc5 4.2.4, httpcore5 5.4.4, httpclient5 5.6.5, JDK 21)
> // I/O dispatch thread blocked in SharedInputBuffer.consumeContent
> "httpclient-dispatch-2" daemon
>    java.lang.Thread.State: WAITING (parking)
>     at 
> org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.consumeContent(SharedInputBuffer.java:101)
>     at 
> org.apache.cxf.transport.http.asyncclient.hc5.CXFHttpAsyncResponseConsumer.consume(CXFHttpAsyncResponseConsumer.java:81)
>     at 
> org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.consume(HttpAsyncMainClientExec.java:275)
>     at 
> org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.consumeData(ClientHttp1StreamHandler.java:270)
>     at 
> org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeData(ClientHttp1StreamDuplexer.java:364)
>     at 
> org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:333)
>     at 
> org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
>     at 
> org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
>     at 
> org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:139)
>     at 
> org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
>     ...
> // Calling thread blocked in SharedInputBuffer.waitForData
> "test-caller" daemon
>    java.lang.Thread.State: WAITING (parking)
>     at 
> org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.waitForData(SharedInputBuffer.java:170)
>     at 
> org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.read(SharedInputBuffer.java:244)
>     at 
> org.apache.cxf.transport.http.asyncclient.hc5.AsyncHTTPConduit$AsyncWrappedOutputStream$6.read(AsyncHTTPConduit.java:734)
>     at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155)
>     at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:385)
>     at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:116)
>     at com.ctc.wstx.io.MergedReader.read(MergedReader.java:104)
>     at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:88)
>     at 
> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
>     ...
> {code}
> In production we saw exactly matching pairs: 8 dispatcher threads at 
> {{SharedInputBuffer.java:101}} and 8 reader threads at 
> {{SharedInputBuffer.java:170}} (via {{read(byte[],int,int)}} at line 244).
> h3. Reproduction
> The attached {{AsyncHTTPConduitLargeResponseHangTest.java}} is a 
> self-contained JUnit 5 test. It needs only CXF jaxws frontend, the http-hc5 
> transport, the JDK {{HttpServer}} and JUnit 5. It starts a local HTTP server 
> that returns a 300 KB SOAP response with {{Content-Length}} and calls it from 
> 20 threads × 5 times through {{JaxWsProxyFactoryBean}} with 
> {{USE_ASYNC=true}}.
> || httpcore5 / httpclient5 || chunkLength not set (16320) || chunkLength = 
> 65536 ||
> | 5.4.3 / 5.6.4 | passes | passes |
> | 5.4.4 / 5.6.5 | *hangs* (all 20 caller threads blocked, test times out 
> after 15 s; reproducible) | passes |
> h3. Workaround
> Set {{HTTPClientPolicy.setChunkLength()}} well above the httpcore5 HTTP/1.1 
> buffer size, e.g. 65536. Since then, the problem has not occurred in our 
> production system.
> h3. Suggested fix
> In {{SharedInputBuffer.consumeContent()}}, signal {{condition}} before 
> blocking on {{suspendInput}}:
> {code:java}
> if (!buffer().hasRemaining() && !this.endOfStream) {
>     this.condition.signalAll();   // wake up a reader waiting on waitingBuffer
>     try {
>         suspendInput.await();
>     } catch (InterruptedException ex) {
>         throw new IOException("Interrupted while waiting buffer to be drained 
> ");
>     }
> }
> this.condition.signalAll();
> {code}
> This fix is untested. Independently of that, it may be worth not blocking the 
> I/O dispatcher thread at all and using {{CapacityChannel}} for flow control 
> instead ({{updateCapacity()}} currently always grants {{Integer.MAX_VALUE}}).
> Attachments:
> - AsyncHTTPConduitLargeResponseHangTest.java
> - threaddump-httpcore5-5.4.4.txt (full dump from the reproduction)



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to