allthingssecurity opened a new pull request, #27758: URL: https://github.com/apache/camel/pull/27758
# Description [CAMEL-25574](https://issues.apache.org/jira/browse/CAMEL-25574) `GzipDeflaterDataFormat.unmarshal` gave the stream it got to `GzipCompressorInputStream` of commons-compress. When the body of the message is an `InputStream`, that stream is the body itself, and with stream caching (on by default) the body is an in-memory stream cache: `InputStreamCache`, which extends `ByteArrayInputStream`, or `ByteArrayInputStreamCache`, which delegates `mark` and `reset` to one. The gzip reader uses a stream with `markSupported()` as it is, and calls `mark` before every buffer of 8192 bytes that it reads, to `reset` and step back when the compressed data ends inside the buffer. So after the unmarshal the mark of the stream cache was behind the 10 byte gzip header, or at the start of the last buffer of a larger body, and `reset()` of the stream cache, which for a `ByteArrayInputStream` goes back to the last mark, no longer went back to the start of the body. Camel resets the stream cache whenever the body is to be read again, and the original message kept for `useOriginalMessage` holds the same stream cache object. With the default configuration: - A dead letter channel with `useOriginalMessage()`, in a route that fails after `unmarshal().gzipDeflater()`, got the original body without its first bytes: the last 21 bytes of a gzip body of 31, the last 3634 of one of 20028. That is not a gzip stream any more, and nothing was logged. - A route that keeps the compressed body (in a property, a variable or a header) and uses it after the unmarshal, for example to archive it, got the same. A body that is a `byte[]` or a file is not affected (a new stream is opened for each read), and neither is a stream cache spooled to disk (`FileInputStreamCache` does not support `mark`, so the reader wraps it; read in the code, not run). This change: `unmarshal` wraps the stream in a `BufferedInputStream` before it gives it to the gzip reader, so the reader marks and resets a stream of its own and the body is only read. For a stream without mark support the reader already wrapped it the same way; for the in-memory stream caches this adds one copy through a buffer of 8192 bytes. The unmarshalled result is the same, so there is no upgrade note. Not changed: the in-memory stream caches. Any reader that calls `mark` in the middle of a stream-cached body moves the position that `StreamCache.reset()` goes back to. There is one more place in the main sources of Camel that moves the mark of such a stream, and it is not changed here: the copy with a buffer size, `IOHelper.copy(InputStream, OutputStream, int, boolean, long)` (`IOHelper.java:224-227`), calls `mark(0)` and `reset()` on any `ByteArrayInputStream` (so on an `InputStreamCache`, not on a `ByteArrayInputStreamCache`) to get its size. That sets the mark to the position the stream has when the copy starts. For a stream that is at its start, which is the usual case because Camel resets the stream cache before each step, nothing changes; for a stream that was already partly read, a later `reset()` goes back to that position and not to the start. Its callers with a stream are `DefaultHttpBinding.copyStream`, `UndertowConsumer` (`useStreaming`) and `CachedOutputStream` (a ne w stream); I did not run a case where one of them gets a partly read body. Whether `reset()` of the stream caches should always go to the start is a question for a separate ticket. Related: the request body stream cache of camel-netty-http (`NettyChannelBufferStreamCache`) has a separate ticket and pull request (its `InputStream` read contract). The two changes are independent: with that change the netty-http stream cache does not support `mark`, so the gzip reader wraps it by itself and neither change needs the other. Tests: - New `GzipDataFormatBodyReadAgainTest` in camel-zip-deflater (3 tests, plain `direct` routes, no option set): a dead letter channel with `useOriginalMessage()` after a failure behind the unmarshal, for a small and for a 20 KB gzip body, and a route that puts the compressed body into a property and reads it after the unmarshal. - Without the change (the committed test class with the `GzipDeflaterDataFormat` of main) all 3 fail: - `deadLetterChannelGetsTheWholeOriginalBody`: `array lengths differ, expected: <31> but was: <21>` - `deadLetterChannelGetsTheWholeOriginalBodyOfALargeMessage`: `array lengths differ, expected: <20028> but was: <3634>` - `theCompressedBodyKeptInAPropertyCanBeReadAfterTheUnmarshal`: `array lengths differ, expected: <31> but was: <21>` - With the change, camel-zip-deflater: 16 tests, 0 failures. Not run: `onException(...).useOriginalMessage()`, and a stream cache spooled to disk. Found with a Lean 4 model of the stream cache (length, position, mark) and of the buffer loop of the gzip reader: the model shows that on main a `reset()` after the unmarshal never goes back before the end of the gzip header, for every size of body, so only the tail is returned (21 of 31 and 3634 of 20028 bytes in the two tested cases). The fix is modelled as not marking the body at all, so for the fix the model says nothing beyond that; the tests above are the evidence for it. I then reproduced it with the data format. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-zip-deflater, including the formatter and import-sort plugins. No generated file changes. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
