[
https://issues.apache.org/jira/browse/CAMEL-25573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen resolved CAMEL-25573.
---------------------------------
Resolution: Fixed
Fixed by https://github.com/apache/camel/pull/27757 (merged to main as
1c8e8061c2b1, 4.24.0).
_Claude Code on behalf of Claus Ibsen_
> camel-netty-http - unmarshal().gzipDeflater() of a request body fails with
> "Input is not in the .gz format", and a byte-by-byte read of the body ends at
> the first 0xFF byte or throws at its end
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25573
> URL: https://issues.apache.org/jira/browse/CAMEL-25573
> Project: Camel
> Issue Type: Bug
> Components: camel-netty-http
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> The body of a request received by a netty-http consumer is a
> {{NettyChannelBufferStreamCache}}
> ({{DefaultNettyHttpBinding.toCamelMessage}}, lines 119-123 of main
> 2a0a2b54f4ba; not with {{disableStreamCache=true}} or {{httpProxy=true}}).
> The class is a {{java.io.InputStream}}, and a route step that asks for the
> body as an {{InputStream}} gets this object. Its {{read()}} does not keep the
> {{InputStream}} contract. With the default configuration:
> # Loud: {{from("netty-http:...").unmarshal().gzipDeflater()}} does not work.
> Every request fails with HTTP 500 and {{java.io.IOException: Input is not in
> the .gz format.}} in the log. The same holds for any reader that reads the
> first bytes one by one (to sniff a header, for example): it gets wrong values
> for bytes from 0x80, or an {{IndexOutOfBoundsException}} at the end of the
> body.
> # Silent, and narrower: user code or a library that reads the body byte by
> byte ({{while ((b = in.read()) != -1)}}) gets only the bytes before the first
> 0xFF, without any error, and negative values for bytes from 0x80. A byte 0xFF
> is common in binary payloads and never occurs in UTF-8 text, so text bodies
> are not cut; for them the loop ends with the {{IndexOutOfBoundsException}}
> (loud).
> Not affected: everything that reads in bulk. The type converters of Camel to
> {{byte[]}} and {{String}} ({{StreamCacheConverter.convertToByteArray}} calls
> {{writeTo}}) and {{NettyHttpConverter.toByteBuf}} read with {{read(byte[],
> int, int)}}, which is why most routes do not see this. Silent loss is not the
> general case.
> The causes in the class:
> * {{read()}} (line 56) is {{return buffer.readByte();}}.
> {{ByteBuf.readByte()}} returns a Java {{byte}} (-128 to 127) and throws
> {{IndexOutOfBoundsException}} when nothing is left. {{InputStream.read()}}
> must return "an int in the range 0 to 255" and -1 at the end of the stream.
> So a byte 0xFF is returned as -1, which every reader takes for the end of the
> stream, bytes 0x80 to 0xFE are returned as negative values, and at the end of
> the body the method throws instead of returning -1.
> * {{markSupported()}} (line 51) returns true, but {{mark(int)}} is not
> overridden (it is the empty method of {{InputStream}}). The gzip reader of
> commons-compress uses a stream with {{markSupported()}} as it is, so it reads
> the magic number with {{read()}} and gets -117 for the byte 0x8b.
> ({{java.util.zip.GZIPInputStream}} fails with
> {{NettyChannelBufferStreamCache.read() returned value out of range -1..255:
> -117}}.)
> * {{read(byte[], int, int)}} (lines 65-78) takes {{buffer.capacity()}} for
> the end of the content; for a buffer whose capacity is larger than its writer
> index it throws {{IndexOutOfBoundsException}} at the end of the content. (The
> aggregated content of a request has no spare capacity, so this one is not
> reached by the consumer today.)
> h3. Reproduction
> New {{NettyHttpGzipRequestBodyTest}} (4 tests, real consumer and producer)
> and {{NettyChannelBufferStreamCacheTest}} (7 tests on the class). With the
> class of main 10 of the 11 fail:
> * {{gzipDataFormatUnmarshalsTheRequestBody}},
> {{resetReturnsTheWholeRequestBodyAfterTheGzipUnmarshal}},
> {{writeToReturnsTheWholeRequestBodyAfterTheGzipUnmarshal}}:
> {{NettyHttpOperationFailedException ... statusCode: 500}}, in the log of the
> consumer {{java.io.IOException: Input is not in the .gz format.}} (all three
> fail at the unmarshal)
> * {{processorReadsTheRequestBodyByteByByte}} (body {{10 20 FF 30 40}}, the
> route counts the bytes with {{read()}}): {{expected: <5> but was: <2>}}
> * {{readReturnsEveryByteAsUnsignedValue}}: {{expected: <128> but was: <-128>}}
> * {{readLoopCopiesBinaryContent}}: {{array lengths differ, expected: <5> but
> was: <2>}}
> * {{readLoopEndsAtTheEndOfText}}: {{IndexOutOfBounds readerIndex(5) +
> length(1) exceeds writerIndex(5)}}
> * {{markIsNotSupported}}: {{expected: <false> but was: <true>}}
> * {{readStopsAtTheLastWrittenByte}}: {{IndexOutOfBounds readerIndex(0) +
> length(16) exceeds writerIndex(3)}}
> * {{gzipInputStreamReadsItsHeader}}: {{IOException:
> ...NettyChannelBufferStreamCache.read() returned value out of range -1..255:
> -117}}
> {{resetAndWriteToReturnTheWholeBodyAfterAReaderThatMarks}} passes on main too
> (there {{mark}} does nothing either); it guards the fix against a
> {{mark(int)}} that moves the reset position, see below.
> h3. Proposed fix
> * {{read()}} returns {{buffer.readByte() & 0xFF}}, and -1 when the buffer has
> no readable byte.
> * {{markSupported()}} returns false. {{mark(int)}} stays the empty method and
> {{reset()}} is not changed: it goes back to the start of the body, as
> {{StreamCache.reset()}} needs. A reader that needs mark support wraps the
> stream (the gzip reader of commons-compress does that by itself with a
> {{BufferedInputStream}}).
> * {{read(byte[], int, int)}} uses {{buffer.readableBytes()}}.
> Why not implement {{mark(int)}}: {{reset()}}, {{writeTo()}} (used by the
> converters to {{byte[]}} and {{String}}) and {{NettyHttpConverter.toByteBuf}}
> all go back to the marked reader index of the buffer, which today is always
> the start because only the constructor marks. If {{mark(int)}} moved it, a
> reader that marks in the middle (the gzip reader marks before every buffer of
> 8192 bytes) would leave the stream cache returning only its tail: with that
> variant the tests got 3634 of 20028 bytes from {{reset()}} and {{writeTo()}}
> after the gzip unmarshal. A stream cache without mark support whose
> {{reset()}} goes back to the start exists already: {{FileInputStreamCache}}
> (line 59, no {{markSupported}} or {{mark}} override; {{reset()}} at line 89
> drops the open stream, so the next read starts at the beginning of the file).
> The in-memory stream caches of camel-support differ: they are, or delegate
> to, a {{ByteArrayInputStream}}, which supports mark.
> Behaviour change: code that reads the body byte by byte now gets values from
> 0 to 255 and -1 at the end, and {{markSupported()}} is false. An upgrade note
> for 4.24 is included (the fix creates {{camel-4x-upgrade-guide-4_24.adoc}}).
> In the main sources of Camel the only caller of {{markSupported()}} on a body
> stream that can be this class is {{BlobClientWrapper}} of
> camel-azure-storage-blob (line 137; read, not run). camel-netty-http with the
> fix: 312 tests, 0 failures, 8 skipped (skipped before as well).
> Not run: a byte-by-byte reader of a text body in a route (the exception at
> the end is shown on the class only), a dead letter channel with
> {{useOriginalMessage}}.
> Related, and independent of this ticket: the gzip data format of
> camel-zip-deflater moving the mark of the in-memory stream caches of
> camel-support has its own ticket. This fix does not rely on that one.
> Found with a Lean 4 model of {{read()}} and of the copy loop of a reader:
> "the reader gets the whole body" fails for every body with a byte 0xFF (the
> reader ends normally with the bytes before it); a body without one ends with
> the exception in the tested case. For the fixed {{read()}} the copy loop is
> proved to return every body, and main and the fix are proved equal for bytes
> below 0x80. For {{reset()}} the model shows that no sequence of read, mark
> and reset calls of a reader changes where it goes back to (by construction:
> {{mark}} does nothing), and gives the 3634 of 20028 bytes for the variant
> with a marking {{mark(int)}}. Then confirmed with the consumer as above.
> Affected: main; camel-4.22.x, camel-4.18.x and camel-4.14.x have the same
> lines (read, not run).
> Duplicate check (2026-10-12): JIRA text "NettyChannelBufferStreamCache" (5:
> CAMEL-7884, CAMEL-8284, CAMEL-8408, CAMEL-13967, CAMEL-14166, none about
> {{read()}} or {{mark}}), "netty-http markSupported" (none), "not in the .gz
> format" (none), "netty-http gzip" (1, unrelated), "netty-http InputStream
> read negative" (1, unrelated); GitHub pull requests
> "NettyChannelBufferStreamCache": 1 merged (#25247, locks), none open.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
