[
https://issues.apache.org/jira/browse/HTTPCLIENT-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18106506#comment-18106506
]
Arturo Bernal commented on HTTPCLIENT-2430:
-------------------------------------------
[~jonnybot]
Thanks for the detailed report. I was able to reproduce the hang with
Brotli-compressed content larger than the decoder input buffer. The decoder
input buffer was not being reset before reuse.
I have added regression coverage for both single-buffer and chunked input and
pushed a fix in the PR.
Could you please test the [PR
877|https://github.com/apache/httpcomponents-client/pull/877] against your
original reproducer and confirm it resolves the issue on your side?
> Brotli decompression fails for large payloads
> ---------------------------------------------
>
> Key: HTTPCLIENT-2430
> URL: https://issues.apache.org/jira/browse/HTTPCLIENT-2430
> Project: HttpComponents HttpClient
> Issue Type: Bug
> Components: HttpClient (async)
> Affects Versions: 5.6.1
> Reporter: Jonny Carter
> Priority: Major
>
> h2. Symptom
> With transparent async content decompression (new in 5.6) and brotli4j on the
> classpath, any response with {{Content-Encoding: br}} whose *compressed* body
> is larger than 8 KB never completes.
> {{InflatingBrotliDataConsumer.consume()}} stops making progress once the
> decoder's fixed 8 KB input buffer is full: the {{xfer == 0}} branch calls
> {{decoder.push(0)}} / {{pump()}} in a loop that can neither advance nor fail,
> so {{while (src.hasRemaining())}} never exits.
> In a real async client the consequences are severe and silent: the exchange
> never completes, no response timeout fires, and the pooled connection is
> never released. Under load the pool drains connection by connection until
> every request to that route hangs. There is no exception and no log line
> anywhere.
> We found this on 5.6.4; the code is unchanged on the 5.6.x branch head.
> HTTPCLIENT-2428 (fixed for 5.7-alpha1) touches the same class but only
> replaces the capacity plumbing; the {{consume()}} loop is untouched, so
> 5.7-alpha1 appears to be affected as well.
> A second, smaller defect in the same class: a single spurious byte after a
> complete, valid brotli stream fails the exchange with {{{}IOException: Brotli
> stream corrupted{}}}, although the content is fully recoverable and the
> trailer could be ignored.
> h2. How to reproduce
> Failing unit tests against {{InflatingBrotliDataConsumerTest}} are attached
> and available on a GitHub branch in my fork
> [https://github.com/jonnybot0/httpcomponents-client/tree/bugfix-brotli]
> (commit 402e689774fe76ae9385e9b2e7b603a3ac11f5bc).
> * {{inflateBrotliLargerThanInputBufferSingleBuffer}} – well-formed brotli
> stream, ~64 KB of incompressible (random) payload so the compressed form
> exceeds 8 KB, delivered as one buffer. Fails with {{{}execution timed out
> after 5000 ms{}}}; the timeout stack pins the busy-loop at
> {{InflatingBrotliDataConsumer.consume}} ({{{}decoder.push{}}} in the {{xfer
> == 0}} branch).
> * {{inflateBrotliLargerThanInputBufferChunked}} – the same stream in 1 KB
> chunks, i.e. identical in shape to the existing passing test; the only
> difference is the compressed size. Same 5 s timeout.
> * {{inflateBrotliWithTrailingByte}} – valid small stream plus one trailing
> byte; fails with {{{}IOException: Brotli stream corrupted{}}}.
> Note the existing happy-path test passes only because its highly compressible
> payload stays well under the 8 KB buffer.
> Core of the repro (the harness drives the consumer directly, no network):
> {code:java}
> final byte[] original = new byte[64 * 1024];
> new java.util.Random(42).nextBytes(original); // incompressible =>
> compressed > 8 KB
> final byte[] compressed = Encoder.compress(original,
> new Encoder.Parameters().setQuality(6).setWindow(22));
> final InflatingBrotliDataConsumer inflating = new
> InflatingBrotliDataConsumer(rawByteCollector);
> inflating.consume(ByteBuffer.wrap(compressed)); // never returns
> inflating.streamEnd(Collections.emptyList());
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
