[
https://issues.apache.org/jira/browse/HTTPCLIENT-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18106699#comment-18106699
]
Jonny Carter edited comment on HTTPCLIENT-2430 at 8/21/26 4:53 PM:
-------------------------------------------------------------------
Hey, [~abernal], thanks for the quick response! The quick checks I'm able to do
in unit tests (both within my codebase and using the one I wrote) largely pass.
The only fail was this test from my report branch:
{code:java}
/**
* A single spurious byte after the end of a valid brotli stream must not
corrupt or hang
* the decoder: the stream is complete, so the decoded content is fully
recoverable.
*/
@Test
void inflateBrotliWithTrailingByte() {
assertTimeoutPreemptively(java.time.Duration.ofSeconds(5), () -> {
final byte[] original = "{\"hello\":
\"world\"}".getBytes(StandardCharsets.UTF_8);
final byte[] compressed = brCompressRaw(original);
final byte[] withTrailing = java.util.Arrays.copyOf(compressed,
compressed.length + 1);
final byte[] inflated = inflate(withTrailing, Integer.MAX_VALUE);
org.junit.jupiter.api.Assertions.assertArrayEquals(original, inflated,
"br inflate mismatch");
});
}{code}
It fails with
{code:java}
java.io.IOException: Brotli stream corrupted
at
org.apache.hc.client5.http.async.methods.InflatingBrotliDataConsumer.pump(InflatingBrotliDataConsumer.java:160)
at
org.apache.hc.client5.http.async.methods.InflatingBrotliDataConsumer.consume(InflatingBrotliDataConsumer.java:99)
at
org.apache.hc.client5.http.async.methods.InflatingBrotliDataConsumerTest.inflate(InflatingBrotliDataConsumerTest.java:284)
....more stacktrace follows{code}
I think that's okay. That test was more the LLM being overly defensive, and I
think "fails fast with IOException" is also an acceptable outcome.
That gives me enough confidence to believe the bugfix you did should address
the root cause. Glad to see you intend to backport to 5.6.5 as well!
was (Author: jonnybot):
Hey, [~abernal], thanks for the quick response! The quick checks I'm able to do
in unit tests (both within my codebase and using the one I wrote) largely pass.
The only fail was this test from my report branch:
{code:java}
/**
* A single spurious byte after the end of a valid brotli stream must not
corrupt or hang
* the decoder: the stream is complete, so the decoded content is fully
recoverable.
*/
@Test
void inflateBrotliWithTrailingByte() {
assertTimeoutPreemptively(java.time.Duration.ofSeconds(5), () -> {
final byte[] original = "{\"hello\":
\"world\"}".getBytes(StandardCharsets.UTF_8);
final byte[] compressed = brCompressRaw(original);
final byte[] withTrailing = java.util.Arrays.copyOf(compressed,
compressed.length + 1);
final byte[] inflated = inflate(withTrailing, Integer.MAX_VALUE);
org.junit.jupiter.api.Assertions.assertArrayEquals(original, inflated,
"br inflate mismatch");
});
}{code}
It fails with
{code:java}
java.io.IOException: Brotli stream corrupted
at
org.apache.hc.client5.http.async.methods.InflatingBrotliDataConsumer.pump(InflatingBrotliDataConsumer.java:160)
at
org.apache.hc.client5.http.async.methods.InflatingBrotliDataConsumer.consume(InflatingBrotliDataConsumer.java:99)
at
org.apache.hc.client5.http.async.methods.InflatingBrotliDataConsumerTest.inflate(InflatingBrotliDataConsumerTest.java:284)
....more stacktrace follows{code}
I think that's okay. That test was more the LLM being overly defensive, and I
think "fails fast with IOException" is also an acceptable outcome.
That gives me enough confidence to believe the root bug should work. Glad to
see you intend to backport to 5.6.5 as well!
> Brotli decompression fails for large payloads
> ---------------------------------------------
>
> Key: HTTPCLIENT-2430
> URL: https://issues.apache.org/jira/browse/HTTPCLIENT-2430
> Project: HttpComponents HttpClient
> Issue Type: Bug
> Components: HttpClient (async)
> Affects Versions: 5.6.1
> Reporter: Jonny Carter
> Assignee: Arturo Bernal
> Priority: Major
> Fix For: 5.6.5, 5.7-alpha2
>
>
> h2. Symptom
> With transparent async content decompression (new in 5.6) and brotli4j on the
> classpath, any response with {{Content-Encoding: br}} whose *compressed* body
> is larger than 8 KB never completes.
> {{InflatingBrotliDataConsumer.consume()}} stops making progress once the
> decoder's fixed 8 KB input buffer is full: the {{xfer == 0}} branch calls
> {{decoder.push(0)}} / {{pump()}} in a loop that can neither advance nor fail,
> so {{while (src.hasRemaining())}} never exits.
> In a real async client the consequences are severe and silent: the exchange
> never completes, no response timeout fires, and the pooled connection is
> never released. Under load the pool drains connection by connection until
> every request to that route hangs. There is no exception and no log line
> anywhere.
> We found this on 5.6.4; the code is unchanged on the 5.6.x branch head.
> HTTPCLIENT-2428 (fixed for 5.7-alpha1) touches the same class but only
> replaces the capacity plumbing; the {{consume()}} loop is untouched, so
> 5.7-alpha1 appears to be affected as well.
> A second, smaller defect in the same class: a single spurious byte after a
> complete, valid brotli stream fails the exchange with {{{}IOException: Brotli
> stream corrupted{}}}, although the content is fully recoverable and the
> trailer could be ignored.
> h2. How to reproduce
> Failing unit tests against {{InflatingBrotliDataConsumerTest}} are attached
> and available on a GitHub branch in my fork
> [https://github.com/jonnybot0/httpcomponents-client/tree/bugfix-brotli]
> (commit 402e689774fe76ae9385e9b2e7b603a3ac11f5bc).
> * {{inflateBrotliLargerThanInputBufferSingleBuffer}} – well-formed brotli
> stream, ~64 KB of incompressible (random) payload so the compressed form
> exceeds 8 KB, delivered as one buffer. Fails with {{{}execution timed out
> after 5000 ms{}}}; the timeout stack pins the busy-loop at
> {{InflatingBrotliDataConsumer.consume}} ({{{}decoder.push{}}} in the {{xfer
> == 0}} branch).
> * {{inflateBrotliLargerThanInputBufferChunked}} – the same stream in 1 KB
> chunks, i.e. identical in shape to the existing passing test; the only
> difference is the compressed size. Same 5 s timeout.
> * {{inflateBrotliWithTrailingByte}} – valid small stream plus one trailing
> byte; fails with {{{}IOException: Brotli stream corrupted{}}}.
> Note the existing happy-path test passes only because its highly compressible
> payload stays well under the 8 KB buffer.
> Core of the repro (the harness drives the consumer directly, no network):
> {code:java}
> final byte[] original = new byte[64 * 1024];
> new java.util.Random(42).nextBytes(original); // incompressible =>
> compressed > 8 KB
> final byte[] compressed = Encoder.compress(original,
> new Encoder.Parameters().setQuality(6).setWindow(22));
> final InflatingBrotliDataConsumer inflating = new
> InflatingBrotliDataConsumer(rawByteCollector);
> inflating.consume(ByteBuffer.wrap(compressed)); // never returns
> inflating.streamEnd(Collections.emptyList());
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
