shashank created CAMEL-25905:
--------------------------------
Summary: camel-file - with checksumFileAlgorithm and charset, the
producer decodes the body with the JVM default charset (non-ASCII replaced) and
writes a checksum that does not match the written file
Key: CAMEL-25905
URL: https://issues.apache.org/jira/browse/CAMEL-25905
Project: Camel
Issue Type: Bug
Components: camel-file
Reporter: shashank
Assignee: shashank
{{FileOperations.storeFile}} (main 74ee085f7d64, lines 349-375) implements
{{checksumFileAlgorithm}} (CAMEL-22846, "calculate a checksum from the file
that has been written") by wrapping the body in a {{DigestInputStream}}:
{code:java}
if (endpoint.getChecksumFileAlgorithm() != null) {
MessageDigest digest =
MessageDigest.getInstance(endpoint.getChecksumFileAlgorithm());
InputStream is = exchange.getMessage().getBody(FileInputStream.class);
if (is == null) {
is = mandatoryBodyStream(exchange, exchange.getMessage(), file);
}
dis = new DigestInputStream(is, digest);
}
if (charset != null) {
handleReaderAsFileSource(exchange, file, charset, dis);
} ...
{code}
and {{handleReaderAsFileSource}} (lines 439-456) skips the {{Reader}}
conversion when it gets that stream and reads it with {{new
InputStreamReader(is)}}, the default charset of the JVM. So adding
{{checksumFileAlgorithm}} to a producer with {{charset}} changes what is
written, and the checksum is that of the body, not of the file:
* the bytes of a file consumed with {{charset=ISO-8859-1}} (or any body whose
bytes are in the exchange charset) are decoded as UTF-8: every non-ASCII
character is written as U+FFFD. Without the checksum option the same route
writes the file right
({{from("file:in?charset=ISO-8859-1").to("file:out?charset=UTF-8&checksumFileAlgorithm=sha256")}}
turns {{Grüße aus Köln}} into {{Gr??e aus K?ln}});
* the checksum (the {{CamelFileChecksum}} header and the {{.sha256}} file) is
calculated over the bytes before re-encoding, so it does not match the written
file as soon as the encoding changes a byte: a receiver that verifies the
checksum file rejects every such file;
* a String body whose exchange carries the endpoint charset (for example sent
with a {{ProducerTemplate}} to
{{file:out?charset=ISO-8859-1&checksumFileAlgorithm=sha256}}) is converted to
ISO-8859-1 bytes, decoded as UTF-8 and fails with
{{UnmappableCharacterException}};
* with {{fileExist=Append}} and {{appendChars}}, the appended characters are
written after the digest stream was read, so the checksum leaves them out.
*Fix*: the digest is updated with the bytes written to the file: a
{{DigestOutputStream}} under the writer of the charset path, and the buffer and
the {{appendChars}} bytes of the stream path. The body is decoded as without
the checksum option (the {{Reader}} conversion with the exchange charset). The
writer of the charset path still reports unmappable characters, as
{{Files.newBufferedWriter}} did. Upgrade note in the 4.24 guide (checksums of
such files differ from the ones written before).
*Tests*: new {{FileProducerChecksumOfWrittenFileTest}} in camel-core (4 tests).
On main: {{transcodedFileHasItsOwnChecksumAndContent}} fails with {{array
lengths differ, expected: <17> but was: <20>}} (three U+FFFD of three bytes
each), {{stringWrittenWithCharsetHasTheChecksumOfTheFile}} fails with
{{CamelExecutionException}} caused by {{UnmappableCharacterException}},
{{appendCharsArePartOfTheChecksum}} fails with {{expected: <d2a84f4b...> but
was: <a591a6d4...>}} (the checksum of {{Hello World}} without the newline).
{{bodyWithoutCharsetKeepsItsChecksum}} is a control and passes on main. The 9
tests of {{FileProducerChecksumFileAlgorithmTest}} pass with the fix. The tests
of the {{org.apache.camel.component.file}} package of camel-core: 478 run, 0
failures, 12 skipped; camel-file: 28 run, 0 failures (one flake of
{{FileProducerNoForcedWritesTest}}, passed on rerun; it does not use the
checksum option).
*How it was found*: comparing the code paths of the file producer for one
option (here: the checksum path replaces the input of the charset path). The
JUnit tests above are the evidence.
*Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x ({{new
DigestInputStream}} and {{new InputStreamReader(is)}} at the same lines; read,
not run). 4.14.x has an older implementation
({{GenericFileProducer.writeChecksumFile}} re-reads the body after the write):
no decoding problem there, but the same checksum of the body instead of the
file.
Not changed: with {{fileExist=Append}} to an existing file, the checksum covers
the bytes this exchange wrote, not the whole file. The remote file producers
(ftp, sftp, scp, smb, azure-files) expose {{checksumFileAlgorithm}} too
(inherited from {{GenericFileEndpoint}}) but since the checksum moved into
{{FileOperations}} (CAMEL-22846, 4.18) they never calculate one, so no header
and no checksum file are written (4.14.x wrote one for them); possible
follow-up.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
