[
https://issues.apache.org/jira/browse/CAMEL-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shashank updated CAMEL-25906:
-----------------------------
Description:
{{checksumFileAlgorithm}} ("calculate a checksum from the file that has been
written ... and store the result in the CamelFileChecksum header";
{{checksumWriteFile}} writes it as {{<file>.<algorithm>}}) is declared on
{{GenericFileEndpoint}}, so it is an option of every file component: file, ftp,
ftps, sftp, mina-sftp, scp, smb and azure-files. CAMEL-20172 (4.3) added it for
"camel-file/ftp": {{GenericFileProducer.writeChecksumFile}} calculated the
checksum from the body for every producer.
CAMEL-22846 (4.18) moved the calculation into {{FileOperations.storeFile}} (a
digest while it writes the local file), and
{{GenericFileProducer.writeChecksumFile}} (main 74ee085f7d64, line 290) now
only writes the header that the operations set:
{code:java}
String hash = exchange.getMessage().getHeader(FileConstants.FILE_CHECKSUM,
String.class);
if (hash != null) {
... operations.storeFileDirectly(checksumFileName, hash) ...
}
{code}
Only {{FileOperations}} sets {{CamelFileChecksum}}. So since 4.18:
* an ftp, ftps, sftp, mina-sftp, scp, smb or azure-files producer with
{{checksumFileAlgorithm}} uploads the file and writes no checksum file and no
header, without an error or a log line. A receiver that waits for
{{<file>.sha256}} (or verifies it) never gets it; an upgrade from 4.14 LTS to
4.18 LTS loses the checksum files silently;
* when an earlier file producer of the route set the header, e.g.
{{.to("file:archive?checksumFileAlgorithm=sha256").to("sftp:...?checksumFileAlgorithm=sha256")}},
the remote producer writes that checksum: the checksum of the archive copy,
which is not the uploaded file as soon as the route changed the body in
between, or the remote producer encodes it with its {{charset}}.
*Fix* (in {{GenericFileProducer}}, one place for all components): before the
write, a {{CamelFileChecksum}} header set earlier is removed; for operations
other than {{FileOperations}} the checksum is calculated of the bytes the
remote operations upload. ftp, ftps, sftp, mina-sftp and smb read the body as
an {{InputStream}} ({{getMandatoryBody(InputStream.class)}}), or through
{{GenericFileHelper.toInputStream}} when the endpoint has a {{charset}}; scp
and azure-files ignore {{charset}} (the upgrade note says not to combine it
with {{checksumFileAlgorithm}} there). A body that gives the same bytes each
time (text, bytes, a file, a stream cache) is read once more for the checksum;
any other body is turned into a stream cache first when it can be (a {{Reader}}
becomes a reader cache), so the checksum does not use it up before the upload.
An {{InputStream}} that can be read once is wrapped in a {{DigestInputStream}}
that the upload reads, so its checksum takes no second pass and no memory; with
a {{charset}} such a stream is kept as a stream cache first (memory, or a spool
file when spooling is enabled). The {{CamelFileChecksum}} header is set once
the file is in place (after the upload and the rename of a temporary file).
Reading a file body twice: for a large file that is a second read from disk,
the price of keeping the fix in one place instead of a digest in the upload
code of each of the seven operations classes. With {{fileExist=Ignore}} and an
existing file nothing is uploaded and no checksum is written, as with the file
component. {{FileOperations}} is unchanged. Upgrade note in the 4.24 guide.
*Tests*: new {{FtpProducerChecksumFileAlgorithmIT}} (7 tests, camel-ftp, the
embedded FTP server of camel-test-infra-ftp, no Docker): the default
({{checksumWriteFile=true}}) with a String body, a one-shot stream body, a
String and a one-shot stream with {{charset=ISO-8859-1}},
{{checksumWriteFile=false}} (header only), and a file producer earlier in the
route. On main all 5 fail: {{Checksum file should exist ==> expected: <true>
but was: <false>}} (3), {{expected: <a591a6d4...> but was: <null>}} (header),
and {{expected: <01285a4c...> but was: <a591a6d4...>}} (the checksum of the
archived {{Hello World}} written for an uploaded {{Bye World}}).
{{testChecksumOfReaderBody}} (a {{StringReader}} body, with and without
{{charset}}) and {{testNoChecksumHeaderWhenTheExistingFileIsKept}}
({{tempFileName}}, {{fileExist=Ignore}}, {{eagerDeleteTargetFile=false}},
existing target) were added in review; they fail on the first version of the
fix ({{Stream closed}}, and a checksum header for a file that was not written).
With the fix: camel-ftp unit tests and FTP producer ITs ({{*Test}},
{{FtpProducer*IT}}, embedded server) 156 run, 0 failures (12 skipped);
camel-core {{org.apache.camel.component.file}} tests 474 run, 0 failures (12
skipped); camel-file 28 run.
*How it was found*: following up the producer paths of
{{checksumFileAlgorithm}} while fixing its charset path in camel-file (separate
ticket): the option is declared on the shared endpoint, the calculation lives
in one subclass of {{GenericFileOperations}}.
*Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (read, not run:
{{writeChecksumFile}} reads only the header, and no remote operations set it).
camel-4.14.x calculated the checksum for every producer.
_Filed with Claude Code on behalf of allthingssecurity._
was:
{{checksumFileAlgorithm}} ("calculate a checksum from the file that has been
written ... and store the result in the CamelFileChecksum header";
{{checksumWriteFile}} writes it as {{<file>.<algorithm>}}) is declared on
{{GenericFileEndpoint}}, so it is an option of every file component: file, ftp,
ftps, sftp, mina-sftp, scp, smb and azure-files. CAMEL-20172 (4.3) added it for
"camel-file/ftp": {{GenericFileProducer.writeChecksumFile}} calculated the
checksum from the body for every producer.
CAMEL-22846 (4.18) moved the calculation into {{FileOperations.storeFile}} (a
digest while it writes the local file), and
{{GenericFileProducer.writeChecksumFile}} (main 74ee085f7d64, line 290) now
only writes the header that the operations set:
{code:java}
String hash = exchange.getMessage().getHeader(FileConstants.FILE_CHECKSUM,
String.class);
if (hash != null) {
... operations.storeFileDirectly(checksumFileName, hash) ...
}
{code}
Only {{FileOperations}} sets {{CamelFileChecksum}}. So since 4.18:
* an ftp, ftps, sftp, mina-sftp, scp, smb or azure-files producer with
{{checksumFileAlgorithm}} uploads the file and writes no checksum file and no
header, without an error or a log line. A receiver that waits for
{{<file>.sha256}} (or verifies it) never gets it; an upgrade from 4.14 LTS to
4.18 LTS loses the checksum files silently;
* when an earlier file producer of the route set the header, e.g.
{{.to("file:archive?checksumFileAlgorithm=sha256").to("sftp:...?checksumFileAlgorithm=sha256")}},
the remote producer writes that checksum: the checksum of the archive copy,
which is not the uploaded file as soon as the route changed the body in
between, or the remote producer encodes it with its {{charset}}.
*Fix* (in {{GenericFileProducer}}, one place for all components): before the
write, a {{CamelFileChecksum}} header set earlier is removed; for operations
other than {{FileOperations}} the checksum is calculated of the bytes the
remote operations upload. All of them read the body as an {{InputStream}}
({{getMandatoryBody(InputStream.class)}}), or through
{{GenericFileHelper.toInputStream}} when the endpoint has a {{charset}}. A
stream body that can be read once is wrapped in a {{DigestInputStream}} that
the upload reads, so its checksum takes no second pass and no memory; with a
{{charset}} such a stream is kept as a stream cache first (memory, or a spool
file when spooling is enabled). Other bodies (a file, bytes, text, a stream
cache) are read once more for the checksum: for a large file that is a second
read from disk, the price of keeping the fix in one place instead of a digest
in the upload code of each of the seven operations classes. With
{{fileExist=Ignore}} and an existing file nothing is uploaded and no checksum
is written, as with the file component. {{FileOperations}} is unchanged.
Upgrade note in the 4.24 guide.
*Tests*: new {{FtpProducerChecksumFileAlgorithmIT}} (5 tests, camel-ftp, the
embedded FTP server of camel-test-infra-ftp, no Docker): the default
({{checksumWriteFile=true}}) with a String body, a one-shot stream body, a
String and a one-shot stream with {{charset=ISO-8859-1}},
{{checksumWriteFile=false}} (header only), and a file producer earlier in the
route. On main all 5 fail: {{Checksum file should exist ==> expected: <true>
but was: <false>}} (3), {{expected: <a591a6d4...> but was: <null>}} (header),
and {{expected: <01285a4c...> but was: <a591a6d4...>}} (the checksum of the
archived {{Hello World}} written for an uploaded {{Bye World}}). With the fix:
camel-ftp unit tests 81 run, 0 failures (9 skipped); camel-ftp producer ITs
({{*Producer*IT}}: 42 FTP and 4 SFTP classes with the embedded servers) 86 run,
0 failures (3 skipped); camel-core {{org.apache.camel.component.file}} tests
474 run, 0 failures (12 skipped); camel-file 28 run.
*How it was found*: following up the producer paths of
{{checksumFileAlgorithm}} while fixing its charset path in camel-file (separate
ticket): the option is declared on the shared endpoint, the calculation lives
in one subclass of {{GenericFileOperations}}.
*Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (read, not run:
{{writeChecksumFile}} reads only the header, and no remote operations set it).
camel-4.14.x calculated the checksum for every producer.
_Filed with Claude Code on behalf of allthingssecurity._
> camel-ftp / sftp / smb / azure-files - checksumFileAlgorithm writes no
> checksum file since 4.18, or the checksum of an earlier file producer's
> payload
> ------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25906
> URL: https://issues.apache.org/jira/browse/CAMEL-25906
> Project: Camel
> Issue Type: Bug
> Components: camel-file, camel-ftp
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> {{checksumFileAlgorithm}} ("calculate a checksum from the file that has been
> written ... and store the result in the CamelFileChecksum header";
> {{checksumWriteFile}} writes it as {{<file>.<algorithm>}}) is declared on
> {{GenericFileEndpoint}}, so it is an option of every file component: file,
> ftp, ftps, sftp, mina-sftp, scp, smb and azure-files. CAMEL-20172 (4.3) added
> it for "camel-file/ftp": {{GenericFileProducer.writeChecksumFile}} calculated
> the checksum from the body for every producer.
> CAMEL-22846 (4.18) moved the calculation into {{FileOperations.storeFile}} (a
> digest while it writes the local file), and
> {{GenericFileProducer.writeChecksumFile}} (main 74ee085f7d64, line 290) now
> only writes the header that the operations set:
> {code:java}
> String hash = exchange.getMessage().getHeader(FileConstants.FILE_CHECKSUM,
> String.class);
> if (hash != null) {
> ... operations.storeFileDirectly(checksumFileName, hash) ...
> }
> {code}
> Only {{FileOperations}} sets {{CamelFileChecksum}}. So since 4.18:
> * an ftp, ftps, sftp, mina-sftp, scp, smb or azure-files producer with
> {{checksumFileAlgorithm}} uploads the file and writes no checksum file and no
> header, without an error or a log line. A receiver that waits for
> {{<file>.sha256}} (or verifies it) never gets it; an upgrade from 4.14 LTS to
> 4.18 LTS loses the checksum files silently;
> * when an earlier file producer of the route set the header, e.g.
> {{.to("file:archive?checksumFileAlgorithm=sha256").to("sftp:...?checksumFileAlgorithm=sha256")}},
> the remote producer writes that checksum: the checksum of the archive copy,
> which is not the uploaded file as soon as the route changed the body in
> between, or the remote producer encodes it with its {{charset}}.
> *Fix* (in {{GenericFileProducer}}, one place for all components): before the
> write, a {{CamelFileChecksum}} header set earlier is removed; for operations
> other than {{FileOperations}} the checksum is calculated of the bytes the
> remote operations upload. ftp, ftps, sftp, mina-sftp and smb read the body as
> an {{InputStream}} ({{getMandatoryBody(InputStream.class)}}), or through
> {{GenericFileHelper.toInputStream}} when the endpoint has a {{charset}}; scp
> and azure-files ignore {{charset}} (the upgrade note says not to combine it
> with {{checksumFileAlgorithm}} there). A body that gives the same bytes each
> time (text, bytes, a file, a stream cache) is read once more for the
> checksum; any other body is turned into a stream cache first when it can be
> (a {{Reader}} becomes a reader cache), so the checksum does not use it up
> before the upload. An {{InputStream}} that can be read once is wrapped in a
> {{DigestInputStream}} that the upload reads, so its checksum takes no second
> pass and no memory; with a {{charset}} such a stream is kept as a stream
> cache first (memory, or a spool file when spooling is enabled). The
> {{CamelFileChecksum}} header is set once the file is in place (after the
> upload and the rename of a temporary file). Reading a file body twice: for a
> large file that is a second read from disk, the price of keeping the fix in
> one place instead of a digest in the upload code of each of the seven
> operations classes. With {{fileExist=Ignore}} and an existing file nothing is
> uploaded and no checksum is written, as with the file component.
> {{FileOperations}} is unchanged. Upgrade note in the 4.24 guide.
> *Tests*: new {{FtpProducerChecksumFileAlgorithmIT}} (7 tests, camel-ftp, the
> embedded FTP server of camel-test-infra-ftp, no Docker): the default
> ({{checksumWriteFile=true}}) with a String body, a one-shot stream body, a
> String and a one-shot stream with {{charset=ISO-8859-1}},
> {{checksumWriteFile=false}} (header only), and a file producer earlier in the
> route. On main all 5 fail: {{Checksum file should exist ==> expected: <true>
> but was: <false>}} (3), {{expected: <a591a6d4...> but was: <null>}} (header),
> and {{expected: <01285a4c...> but was: <a591a6d4...>}} (the checksum of the
> archived {{Hello World}} written for an uploaded {{Bye World}}).
> {{testChecksumOfReaderBody}} (a {{StringReader}} body, with and without
> {{charset}}) and {{testNoChecksumHeaderWhenTheExistingFileIsKept}}
> ({{tempFileName}}, {{fileExist=Ignore}}, {{eagerDeleteTargetFile=false}},
> existing target) were added in review; they fail on the first version of the
> fix ({{Stream closed}}, and a checksum header for a file that was not
> written). With the fix: camel-ftp unit tests and FTP producer ITs ({{*Test}},
> {{FtpProducer*IT}}, embedded server) 156 run, 0 failures (12 skipped);
> camel-core {{org.apache.camel.component.file}} tests 474 run, 0 failures (12
> skipped); camel-file 28 run.
> *How it was found*: following up the producer paths of
> {{checksumFileAlgorithm}} while fixing its charset path in camel-file
> (separate ticket): the option is declared on the shared endpoint, the
> calculation lives in one subclass of {{GenericFileOperations}}.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (read, not run:
> {{writeChecksumFile}} reads only the header, and no remote operations set
> it). camel-4.14.x calculated the checksum for every producer.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
