This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 713f5208b1a2 CAMEL-25948: camel-mina-sftp - delete the local work file
of a consumed file when the exchange is done (#28466)
713f5208b1a2 is described below
commit 713f5208b1a2f7524d3d73fd7fc1d00c9a69d1b3
Author: allthingssecurity <[email protected]>
AuthorDate: Tue Oct 27 13:48:51 2026 +0530
CAMEL-25948: camel-mina-sftp - delete the local work file of a consumed
file when the exchange is done (#28466)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
.../camel/catalog/docs/mina-sftp-component.adoc | 72 ++++++++
.../src/main/docs/mina-sftp-component.adoc | 72 ++++++++
.../file/remote/mina/MinaSftpOperations.java | 10 ++
.../MinaSftpConsumerLocalWorkDirectoryIT.java | 183 +++++++++++++++++++++
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 11 ++
5 files changed, 348 insertions(+)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/mina-sftp-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/mina-sftp-component.adoc
index c574abf5eb14..8f282d231c19 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/mina-sftp-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/mina-sftp-component.adoc
@@ -318,6 +318,78 @@ YAML::
----
====
+=== Using Local Work Directory
+
+Camel supports consuming from remote SFTP servers and downloading the
+files directly into a local work directory, with the `localWorkDirectory`
option.
+This avoids reading the entire remote file content into memory as it is
+streamed directly into the local file using `FileOutputStream`.
+
+Camel will store to a local file with the same name as the remote file,
+though with `.inprogress` as the extension while the file is being
+downloaded. Afterward, the file is renamed to remove the `.inprogress`
+suffix. And finally, when the Exchange is done (also when it failed),
+the local file is deleted. If the download itself fails, the partly
+written `.inprogress` file is deleted at once.
+
+So if you want to download files from a remote SFTP server and store them
+as files, then you need to route to a file endpoint such as:
+
+[tabs]
+====
+Java::
++
+[source,java]
+----
+from("mina-sftp://[email protected]/download?password=secret&localWorkDirectory=/tmp";)
+ .to("file:inbox");
+----
+
+XML::
++
+[source,xml]
+----
+<route>
+ <from
uri="mina-sftp://[email protected]/download?password=secret&localWorkDirectory=/tmp"/>
+ <to uri="file:inbox"/>
+</route>
+----
+
+YAML::
++
+[source,yaml]
+----
+- route:
+ from:
+ uri: mina-sftp://[email protected]/download
+ parameters:
+ password: secret
+ localWorkDirectory: /tmp
+ steps:
+ - to:
+ uri: file:inbox
+----
+====
+
+The `java.io.File` handle of the local work file is the message body.
+The first file producer of the route uses a `java.io.File.rename` to move
+the local work file to its target, instead of a file copy, as the work file
+is meant to be deleted anyway.
+
+IMPORTANT: After that rename, the message body still names the local work
file, which is gone.
+A later step that reads the file again fails, such as a second file producer
or a `convertBodyTo`.
+Another exchange that keeps the `java.io.File` body, for example from a
`wireTap` or an `aggregate`,
+finds the file deleted once the original exchange is done.
+Convert the body before such a step, for example with
`convertBodyTo(byte[].class)`, which reads the file into memory.
+
+[source,java]
+----
+from("mina-sftp://[email protected]/download?password=secret&localWorkDirectory=/tmp";)
+ .convertBodyTo(byte[].class)
+ .to("file:inbox")
+ .to("file:archive");
+----
+
== Error Handling
=== Connection Retry
diff --git a/components/camel-mina-sftp/src/main/docs/mina-sftp-component.adoc
b/components/camel-mina-sftp/src/main/docs/mina-sftp-component.adoc
index c574abf5eb14..8f282d231c19 100644
--- a/components/camel-mina-sftp/src/main/docs/mina-sftp-component.adoc
+++ b/components/camel-mina-sftp/src/main/docs/mina-sftp-component.adoc
@@ -318,6 +318,78 @@ YAML::
----
====
+=== Using Local Work Directory
+
+Camel supports consuming from remote SFTP servers and downloading the
+files directly into a local work directory, with the `localWorkDirectory`
option.
+This avoids reading the entire remote file content into memory as it is
+streamed directly into the local file using `FileOutputStream`.
+
+Camel will store to a local file with the same name as the remote file,
+though with `.inprogress` as the extension while the file is being
+downloaded. Afterward, the file is renamed to remove the `.inprogress`
+suffix. And finally, when the Exchange is done (also when it failed),
+the local file is deleted. If the download itself fails, the partly
+written `.inprogress` file is deleted at once.
+
+So if you want to download files from a remote SFTP server and store them
+as files, then you need to route to a file endpoint such as:
+
+[tabs]
+====
+Java::
++
+[source,java]
+----
+from("mina-sftp://[email protected]/download?password=secret&localWorkDirectory=/tmp";)
+ .to("file:inbox");
+----
+
+XML::
++
+[source,xml]
+----
+<route>
+ <from
uri="mina-sftp://[email protected]/download?password=secret&localWorkDirectory=/tmp"/>
+ <to uri="file:inbox"/>
+</route>
+----
+
+YAML::
++
+[source,yaml]
+----
+- route:
+ from:
+ uri: mina-sftp://[email protected]/download
+ parameters:
+ password: secret
+ localWorkDirectory: /tmp
+ steps:
+ - to:
+ uri: file:inbox
+----
+====
+
+The `java.io.File` handle of the local work file is the message body.
+The first file producer of the route uses a `java.io.File.rename` to move
+the local work file to its target, instead of a file copy, as the work file
+is meant to be deleted anyway.
+
+IMPORTANT: After that rename, the message body still names the local work
file, which is gone.
+A later step that reads the file again fails, such as a second file producer
or a `convertBodyTo`.
+Another exchange that keeps the `java.io.File` body, for example from a
`wireTap` or an `aggregate`,
+finds the file deleted once the original exchange is done.
+Convert the body before such a step, for example with
`convertBodyTo(byte[].class)`, which reads the file into memory.
+
+[source,java]
+----
+from("mina-sftp://[email protected]/download?password=secret&localWorkDirectory=/tmp";)
+ .convertBodyTo(byte[].class)
+ .to("file:inbox")
+ .to("file:archive");
+----
+
== Error Handling
=== Connection Retry
diff --git
a/components/camel-mina-sftp/src/main/java/org/apache/camel/component/file/remote/mina/MinaSftpOperations.java
b/components/camel-mina-sftp/src/main/java/org/apache/camel/component/file/remote/mina/MinaSftpOperations.java
index dac17391bf29..7b055685059c 100644
---
a/components/camel-mina-sftp/src/main/java/org/apache/camel/component/file/remote/mina/MinaSftpOperations.java
+++
b/components/camel-mina-sftp/src/main/java/org/apache/camel/component/file/remote/mina/MinaSftpOperations.java
@@ -1265,9 +1265,19 @@ public class MinaSftpOperations implements
RemoteFileOperations<SftpRemoteFile>
// create parent folders
temp.getParentFile().mkdirs();
+ // the consumer deletes the local work file named by this header
when the exchange is done, and the file
+ // producer moves it to the target instead of copying it
+ exchange.getIn().setHeader(FtpConstants.FILE_LOCAL_WORK_PATH,
local.getPath());
+
try (InputStream is = sftpClient.read(name);
OutputStream os = new FileOutputStream(temp)) {
IOHelper.copy(is, os);
+ } catch (IOException e) {
+ // the streams are closed by now, so the partly written file
can be deleted, as camel-sftp does
+ if (temp.exists() && !FileUtil.deleteFile(temp)) {
+ LOG.warn("Cannot delete local work file: {} after
retrieving file: {} failed", temp, name);
+ }
+ throw e;
}
LOG.debug("Retrieve file to local work file result: true");
diff --git
a/components/camel-mina-sftp/src/test/java/org/apache/camel/component/file/remote/mina/integration/MinaSftpConsumerLocalWorkDirectoryIT.java
b/components/camel-mina-sftp/src/test/java/org/apache/camel/component/file/remote/mina/integration/MinaSftpConsumerLocalWorkDirectoryIT.java
new file mode 100644
index 000000000000..8efe8f8d4eb7
--- /dev/null
+++
b/components/camel-mina-sftp/src/test/java/org/apache/camel/component/file/remote/mina/integration/MinaSftpConsumerLocalWorkDirectoryIT.java
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.file.remote.mina.integration;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.NotifyBuilder;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.test.junit6.TestSupport;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIf;
+import org.junit.jupiter.api.io.TempDir;
+
+import static org.awaitility.Awaitility.await;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The local work file of a consumed file is deleted when the exchange is
done, as with the sftp, ftp, smb and
+ * azure-files components.
+ */
+@EnabledIf(value =
"org.apache.camel.test.infra.ftp.services.embedded.SftpUtil#hasRequiredAlgorithms('src/test/resources/hostkey.pem')")
+class MinaSftpConsumerLocalWorkDirectoryIT extends MinaSftpServerTestSupport {
+
+ @TempDir
+ Path testDirectory;
+
+ @Override
+ protected void setupResources() throws Exception {
+ service.getFtpRootDir().toFile().mkdirs();
+ }
+
+ private String consumerUri(String fileName) {
+ // a long delay so that a rolled back file is not consumed again while
the test checks the local work directory
+ return "mina-sftp://localhost:"; + service.getPort() + "/" +
service.getFtpRootDir()
+ +
"?username=admin&password=admin&strictHostKeyChecking=no&useUserKnownHostsFile=false"
+ + "&fileName=" + fileName +
"&noop=true&initialDelay=0&delay=600000"
+ + "&localWorkDirectory=" + testDirectory.resolve("lwd");
+ }
+
+ @Test
+ void localWorkFileIsDeletedWhenTheExchangeCompletes() throws Exception {
+ Files.writeString(ftpFile("hello.txt"), "Hello World");
+
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("Hello World");
+
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ from(consumerUri("hello.txt"))
+ .process(this::assertBodyIsTheLocalWorkFile)
+ .to("mock:result");
+ }
+
+ private void assertBodyIsTheLocalWorkFile(Exchange exchange) {
+ File body = exchange.getIn().getBody(File.class);
+ assertEquals(testDirectory.resolve("lwd/hello.txt").toFile(),
body);
+ assertTrue(body.exists(), "The local work file should exist
while the exchange is routed");
+ }
+ });
+
+ mock.assertIsSatisfied();
+
+ await().atMost(10, TimeUnit.SECONDS)
+ .untilAsserted(() ->
assertFalse(Files.exists(testDirectory.resolve("lwd/hello.txt")),
+ "The local work file should be deleted when the
exchange is done"));
+ }
+
+ @Test
+ void localWorkFileIsMovedToTheTargetOfAFileProducer() throws Exception {
+ Files.writeString(ftpFile("moved.txt"), "Moved World");
+
+ NotifyBuilder completed = new
NotifyBuilder(context).whenCompleted(1).create();
+
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ from(consumerUri("moved.txt"))
+ .to(TestSupport.fileUri(testDirectory, "out"))
+ .process(this::failIfTheLocalWorkFileIsLeft);
+ }
+
+ private void failIfTheLocalWorkFileIsLeft(Exchange exchange) {
+ // checked before the exchange is done, so before the consumer
deletes the local work file: a file
+ // producer that copied it would leave it here, and the
exchange fails instead of completing
+ if (Files.exists(testDirectory.resolve("lwd/moved.txt"))) {
+ throw new IllegalStateException("The file producer should
move the local work file, not copy it");
+ }
+ }
+ });
+
+ assertTrue(completed.matches(10, TimeUnit.SECONDS), "The exchange
should complete");
+
+ assertEquals("Moved World",
Files.readString(testDirectory.resolve("out/moved.txt")));
+ }
+
+ @Test
+ void aSecondFileProducerFindsTheLocalWorkFileMoved() throws Exception {
+ // as with camel-sftp: the first file producer moves the local work
file, which the body still names
+ Files.writeString(ftpFile("twice.txt"), "Twice World");
+
+ NotifyBuilder failed = new
NotifyBuilder(context).whenFailed(1).create();
+
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ from(consumerUri("twice.txt"))
+ .to(TestSupport.fileUri(testDirectory, "first"))
+ .to(TestSupport.fileUri(testDirectory, "second"));
+ }
+ });
+
+ assertTrue(failed.matches(10, TimeUnit.SECONDS), "The second file
producer should fail");
+
+ assertEquals("Twice World",
Files.readString(testDirectory.resolve("first/twice.txt")));
+ assertFalse(Files.exists(testDirectory.resolve("second/twice.txt")));
+ }
+
+ @Test
+ void aBodyConvertedBeforeTheFileProducerCanBeWrittenTwice() throws
Exception {
+ // control, which passes without the fix too: the workaround of the
upgrade guide
+ Files.writeString(ftpFile("converted.txt"), "Converted World");
+
+ NotifyBuilder done = new
NotifyBuilder(context).whenCompleted(1).create();
+
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ from(consumerUri("converted.txt"))
+ .convertBodyTo(byte[].class)
+ .to(TestSupport.fileUri(testDirectory, "first"))
+ .to(TestSupport.fileUri(testDirectory, "second"));
+ }
+ });
+
+ assertTrue(done.matches(10, TimeUnit.SECONDS), "The exchange should
complete");
+
+ assertEquals("Converted World",
Files.readString(testDirectory.resolve("first/converted.txt")));
+ assertEquals("Converted World",
Files.readString(testDirectory.resolve("second/converted.txt")));
+ }
+
+ @Test
+ void localWorkFileIsDeletedWhenTheExchangeFails() throws Exception {
+ Files.writeString(ftpFile("bye.txt"), "Bye World");
+
+ NotifyBuilder failed = new
NotifyBuilder(context).whenFailed(1).create();
+
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ from(consumerUri("bye.txt"))
+ .throwException(new IllegalStateException("Forced"));
+ }
+ });
+
+ assertTrue(failed.matches(10, TimeUnit.SECONDS), "The exchange should
fail");
+
+ await().atMost(10, TimeUnit.SECONDS)
+ .untilAsserted(() ->
assertFalse(Files.exists(testDirectory.resolve("lwd/bye.txt")),
+ "The local work file should be deleted when the
exchange fails"));
+ }
+}
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index 0be69c592ec1..b5233ff44e10 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -360,6 +360,17 @@ as the body, so `+${body}+` was the file found. A
`fileName` that uses the file
before. A `fileName` that read the file found through its headers, such as
`${header.CamelFileName}`, must use the file
functions instead (`${file:name}`).
+=== camel-mina-sftp - the local work file is deleted when the exchange is done
+
+With `localWorkDirectory`, camel-mina-sftp now deletes the local work file of
a consumed file when the exchange is done,
+also when it failed, as camel-sftp and camel-ftp do. Before, every consumed
file stayed in the local work directory. A
+file producer in the route now moves the local work file to its target instead
of copying it, while the message body
+still names the local work file. So a later step that reads the body again,
such as a second file producer or a
+`convertBodyTo`, fails because the file is gone, and another exchange that
keeps the `java.io.File` body (for example
+`wireTap`, `seda` without waiting, or `aggregate`) finds it deleted. Convert
the body before the first file producer
+or before handing it on, for example with `convertBodyTo(byte[].class)` (which
reads the file into memory). Routes that
+cleaned the local work directory themselves can stop doing so.
+
=== camel-openstack - potential breaking change
The four remaining Exchange header constants of `SwiftConstants` have been
renamed to follow the Camel naming
