This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new f2c17c00443f CAMEL-26656: camel-sftp - check whether a file exists
with one lstat instead of listing the directory (#29273)
f2c17c00443f is described below
commit f2c17c00443f56660be717e9988537573ecbb664
Author: Federico Mariani <[email protected]>
AuthorDate: Fri Nov 6 17:08:24 2026 +0100
CAMEL-26656: camel-sftp - check whether a file exists with one lstat
instead of listing the directory (#29273)
* CAMEL-26656: camel-sftp - check whether a file exists with one lstat
instead of listing the directory
* CAMEL-26656: camel-sftp - fastExistsCheck lists the directory when the
server does not answer the lstat
* CAMEL-26656: camel-sftp - list the directory on any lstat status other
than no such file
* CAMEL-26656: camel-sftp - say in the lstat comment that only the last
separator is sent as '/'
* CAMEL-26656: camel-sftp - rewrap the long line of the upgrade guide entry
* CAMEL-26656: camel-sftp - getFileAttributes returns the size and
modification time instead of the JSch SftpATTRS
* CAMEL-26656: camel-sftp - test that a connection lost during the lstat
fails the check
Co-authored-by: Claude <[email protected]>
---
.../apache/camel/catalog/docs/sftp-component.adoc | 18 +
.../camel-ftp/src/main/docs/sftp-component.adoc | 18 +
.../component/file/remote/SftpOperations.java | 148 ++++++--
.../strategy/SftpExclusiveReadLockCheck.java | 46 ++-
.../remote/sftp/integration/SftpExistsFileIT.java | 421 +++++++++++++++++++++
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 20 +
6 files changed, 622 insertions(+), 49 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/sftp-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/sftp-component.adoc
index 4c4e55ba2bff..c611cb9ac867 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/sftp-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/sftp-component.adoc
@@ -362,6 +362,24 @@ consumer is created, also with `poll` and `pollEnrich`,
with `IllegalArgumentExc
The `readLock` option is not checked when a custom `exclusiveReadLockStrategy`
or `processStrategy`
is set, as it does not apply then.
+=== Checking whether a file exists
+
+The `sftp` component checks whether a remote file exists with a single `lstat`
request, whatever the number of files
+in the directory. This applies to `fileExist=Ignore`, `Fail` and `Move`, to
`tempFileName` and `tempPrefix`, to a
+producer `doneFileName`, and to `readLock=changed`, which reads the size and
modification time of the file with the same
+request on every check. So the `fastExistsCheck` option no longer makes a
difference for these checks with `sftp`.
+
+The answer comes from the server for the path itself:
+
+* A symbolic link is not followed, as in a directory listing. A dangling link
exists, and `readLock=changed` compares
+ the size and modification time of the link, not those of its target.
+* A server on a case-insensitive file system, such as one backed by Windows or
macOS, finds a file whose name differs
+ only in case, which the upload would replace.
+
+When the server answers the request with another status than "no such file",
for example because it does not permit or
+support it on the path, the component lists the directory instead and compares
the names, as before. This listing is
+the same with and without `fastExistsCheck`.
+
== More Information
For more information, you can look at the xref:ftp-component.adoc[FTP
component].
diff --git a/components/camel-ftp/src/main/docs/sftp-component.adoc
b/components/camel-ftp/src/main/docs/sftp-component.adoc
index 4c4e55ba2bff..c611cb9ac867 100644
--- a/components/camel-ftp/src/main/docs/sftp-component.adoc
+++ b/components/camel-ftp/src/main/docs/sftp-component.adoc
@@ -362,6 +362,24 @@ consumer is created, also with `poll` and `pollEnrich`,
with `IllegalArgumentExc
The `readLock` option is not checked when a custom `exclusiveReadLockStrategy`
or `processStrategy`
is set, as it does not apply then.
+=== Checking whether a file exists
+
+The `sftp` component checks whether a remote file exists with a single `lstat`
request, whatever the number of files
+in the directory. This applies to `fileExist=Ignore`, `Fail` and `Move`, to
`tempFileName` and `tempPrefix`, to a
+producer `doneFileName`, and to `readLock=changed`, which reads the size and
modification time of the file with the same
+request on every check. So the `fastExistsCheck` option no longer makes a
difference for these checks with `sftp`.
+
+The answer comes from the server for the path itself:
+
+* A symbolic link is not followed, as in a directory listing. A dangling link
exists, and `readLock=changed` compares
+ the size and modification time of the link, not those of its target.
+* A server on a case-insensitive file system, such as one backed by Windows or
macOS, finds a file whose name differs
+ only in case, which the upload would replace.
+
+When the server answers the request with another status than "no such file",
for example because it does not permit or
+support it on the path, the component lists the directory instead and compares
the names, as before. This listing is
+the same with and without `fastExistsCheck`.
+
== More Information
For more information, you can look at the xref:ftp-component.adoc[FTP
component].
diff --git
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/SftpOperations.java
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/SftpOperations.java
index bdc6887a6c04..d341e3892170 100644
---
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/SftpOperations.java
+++
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/SftpOperations.java
@@ -45,6 +45,7 @@ import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Proxy;
import com.jcraft.jsch.Session;
+import com.jcraft.jsch.SftpATTRS;
import com.jcraft.jsch.SftpException;
import com.jcraft.jsch.SocketFactory;
import com.jcraft.jsch.UIKeyboardInteractive;
@@ -1413,65 +1414,142 @@ public class SftpOperations
if (endpoint.isFastExistsCheck()) {
return fastExistsFile(name);
}
- // check whether a file already exists
- String directory = FileUtil.onlyPath(name);
- if (directory == null) {
- // assume current dir if no path could be extracted
- directory = ".";
- }
- String onlyName = FileUtil.stripPath(name);
+ return getFileAttributes(name) != null;
+ } finally {
+ lock.unlock();
+ }
+ }
+ /**
+ * Gets the attributes of a file with a single request, whatever the
number of files in its directory.
+ * <p/>
+ * These are the attributes of the path itself, as in a directory listing:
a symbolic link is not followed, so for a
+ * link they are the size and modification time of the link, and a
dangling link exists. When the server answers
+ * that request with another status than no such file, such as when it
does not permit or support it on the path,
+ * the attributes are taken from a listing of the directory of the file
instead.
+ *
+ * @param name the name of the file, with
its directory if any
+ * @return the attributes, or {@code
null} if there is no such file
+ * @throws GenericFileOperationFailedException if the attributes cannot be
read
+ */
+ public FileAttributes getFileAttributes(String name) throws
GenericFileOperationFailedException {
+ lock.lock();
+ try {
+ SftpATTRS attrs;
try {
- @SuppressWarnings("rawtypes")
- List files = channel.ls(directory);
- // can return either null or an empty list depending on FTP
servers
- if (files == null) {
- return false;
- }
- for (Object file : files) {
- ChannelSftp.LsEntry entry = (ChannelSftp.LsEntry) file;
- String existing = entry.getFilename();
- LOG.trace("Existing file: {}, target file: {}", existing,
name);
- existing = FileUtil.stripPath(existing);
- if (existing != null && existing.equals(onlyName)) {
- return true;
- }
- }
- return false;
+ attrs = lstat(name);
} catch (SftpException e) {
- // or an exception can be thrown with id 2 which means file
does not
- // exists
- if (ChannelSftp.SSH_FX_NO_SUCH_FILE == e.id) {
- return false;
+ if (!isLstatUnanswered(e)) {
+ throw new
GenericFileOperationFailedException(e.getMessage(), e);
}
- // otherwise its a more serious error so rethrow
- throw new GenericFileOperationFailedException(e.getMessage(),
e);
+ LOG.debug("Cannot get the attributes of: {} due to: {}.
Listing its directory instead.", name,
+ e.getMessage());
+ attrs = listFileAttributes(name);
}
+ return attrs != null ? new FileAttributes(attrs.getSize(),
attrs.getMTime() * 1000L) : null;
} finally {
lock.unlock();
}
}
- protected boolean fastExistsFile(String name) throws
GenericFileOperationFailedException {
- LOG.trace("fastExistsFile({})", name);
+ /**
+ * The attributes of a file read by {@link #getFileAttributes(String)}.
+ *
+ * @param fileLength the file size
+ * @param lastModified the file modification timestamp (in millis)
+ */
+ public record FileAttributes(long fileLength, long lastModified) {
+ }
+
+ /**
+ * Gets the attributes of a file with a single LSTAT request.
+ *
+ * @param name the name of the file, with its directory if any
+ * @return the attributes, or {@code null} if there is no
such file
+ * @throws SftpException if the server answers with another status, or the
request fails
+ */
+ private SftpATTRS lstat(String name) throws SftpException {
+ // the name starts after the last separator, either '/' or '\\', as in
the store and in the listing of the
+ // directory, and is joined to the directory with '/'. The directory
itself is sent as it is given
+ String directory = FileUtil.onlyPath(name);
+ String onlyName = FileUtil.stripPath(name);
+ String path;
+ if (directory == null) {
+ path = onlyName;
+ } else if (directory.endsWith("/")) {
+ path = directory + onlyName;
+ } else {
+ path = directory + "/" + onlyName;
+ }
+ try {
+ return channel.lstat(quotePathForJsch(path));
+ } catch (SftpException e) {
+ if (ChannelSftp.SSH_FX_NO_SUCH_FILE == e.id) {
+ return null;
+ }
+ throw e;
+ }
+ }
+
+ /**
+ * Whether the server answered an LSTAT request with a status that does
not tell whether the path exists, so that
+ * the directory is listed instead, as before: any status other than no
such file, which {@link #lstat(String)}
+ * already answers. A server may not permit or support the request on a
path that it lists, answer with the generic
+ * failure for a path that does not exist, or send a status of a later
version of the protocol.
+ */
+ private static boolean isLstatUnanswered(SftpException e) {
+ // JSch reports a status sent by the server as an SftpException
without a cause, and wraps an I/O or session
+ // failure as an SftpException (SSH_FX_FAILURE) with that failure as
its cause
+ return e.getCause() == null;
+ }
+
+ /**
+ * Gets the attributes of a file from a listing of its directory,
comparing the names of the listed files.
+ */
+ private SftpATTRS listFileAttributes(String name) throws
GenericFileOperationFailedException {
+ String directory = FileUtil.onlyPath(name);
+ if (directory == null) {
+ // assume current dir if no path could be extracted
+ directory = ".";
+ }
+ String onlyName = FileUtil.stripPath(name);
+
try {
@SuppressWarnings("rawtypes")
- List files = channel.ls(name);
+ List files = channel.ls(directory);
+ // can return either null or an empty list depending on FTP servers
if (files == null) {
- return false;
+ return null;
+ }
+ for (Object file : files) {
+ ChannelSftp.LsEntry entry = (ChannelSftp.LsEntry) file;
+ String existing = entry.getFilename();
+ LOG.trace("Existing file: {}, target file: {}", existing,
name);
+ existing = FileUtil.stripPath(existing);
+ if (existing != null && existing.equals(onlyName)) {
+ return entry.getAttrs();
+ }
}
- return !files.isEmpty();
+ return null;
} catch (SftpException e) {
// or an exception can be thrown with id 2 which means file does
not
// exists
if (ChannelSftp.SSH_FX_NO_SUCH_FILE == e.id) {
- return false;
+ return null;
}
// otherwise its a more serious error so rethrow
throw new GenericFileOperationFailedException(e.getMessage(), e);
}
}
+ protected boolean fastExistsFile(String name) throws
GenericFileOperationFailedException {
+ LOG.trace("fastExistsFile({})", name);
+ // the same as without fastExistsCheck, also when the server does not
answer the LSTAT: JSch ls(name) starts
+ // with a STAT of the path, which such a server would most likely not
answer either, and then lists the whole
+ // directory of an existing file anyway
+ return getFileAttributes(name) != null;
+ }
+
@Override
public boolean sendNoop() throws GenericFileOperationFailedException {
lock.lock();
diff --git
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/SftpExclusiveReadLockCheck.java
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/SftpExclusiveReadLockCheck.java
index aef6ca884e3e..a6069c383ae6 100644
---
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/SftpExclusiveReadLockCheck.java
+++
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/SftpExclusiveReadLockCheck.java
@@ -20,6 +20,7 @@ package org.apache.camel.component.file.remote.strategy;
import com.jcraft.jsch.ChannelSftp;
import org.apache.camel.component.file.GenericFile;
import org.apache.camel.component.file.GenericFileOperations;
+import org.apache.camel.component.file.remote.SftpOperations;
import org.apache.camel.component.file.remote.SftpRemoteFile;
import org.apache.camel.util.StopWatch;
import org.slf4j.Logger;
@@ -52,21 +53,34 @@ public class SftpExclusiveReadLockCheck {
long newLastModified = 0;
long newLength = 0;
- Object[] files = listFiles(operations, file);
-
- LOG.trace("List files {} found {} files", file.getAbsoluteFilePath(),
files.length);
- for (Object f : files) {
- SftpRemoteFile rf = (SftpRemoteFile) f;
- boolean match;
- if (fastExistsCheck) {
- match = rf.getFilename().equals(file.getAbsoluteFilePath())
- || rf.getFilename().equals(file.getFileNameOnly());
- } else {
- match = rf.getFilename().equals(file.getFileNameOnly());
+ SftpOperations sftp = asSftpOperations(operations);
+ if (sftp != null) {
+ // a single request whatever the number of files in the directory.
As in the directory listing, these are the
+ // attributes of the path itself: for a symbolic link, the size
and modification time of the link, not of
+ // its target
+ SftpOperations.FileAttributes attrs =
sftp.getFileAttributes(file.getAbsoluteFilePath());
+ LOG.trace("Attributes of {}: {}", file.getAbsoluteFilePath(),
attrs);
+ if (attrs != null) {
+ newLastModified = attrs.lastModified();
+ newLength = attrs.fileLength();
}
- if (match) {
- newLastModified = rf.getLastModified();
- newLength = rf.getFileLength();
+ } else {
+ Object[] files = listFiles(operations, file);
+
+ LOG.trace("List files {} found {} files",
file.getAbsoluteFilePath(), files.length);
+ for (Object f : files) {
+ SftpRemoteFile rf = (SftpRemoteFile) f;
+ boolean match;
+ if (fastExistsCheck) {
+ match = rf.getFilename().equals(file.getAbsoluteFilePath())
+ || rf.getFilename().equals(file.getFileNameOnly());
+ } else {
+ match = rf.getFilename().equals(file.getFileNameOnly());
+ }
+ if (match) {
+ newLastModified = rf.getLastModified();
+ newLength = rf.getFileLength();
+ }
}
}
@@ -85,6 +99,10 @@ public class SftpExclusiveReadLockCheck {
return false;
}
+ private static SftpOperations asSftpOperations(GenericFileOperations<?>
operations) {
+ return operations instanceof SftpOperations sftp ? sftp : null;
+ }
+
private Object[] listFiles(
GenericFileOperations<ChannelSftp.LsEntry> operations,
GenericFile<ChannelSftp.LsEntry> file) {
if (fastExistsCheck) {
diff --git
a/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/sftp/integration/SftpExistsFileIT.java
b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/sftp/integration/SftpExistsFileIT.java
new file mode 100644
index 000000000000..0993215b878a
--- /dev/null
+++
b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/sftp/integration/SftpExistsFileIT.java
@@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.file.remote.sftp.integration;
+
+import java.io.IOException;
+import java.nio.file.AccessDeniedException;
+import java.nio.file.FileAlreadyExistsException;
+import java.nio.file.FileSystemException;
+import java.nio.file.Files;
+import java.nio.file.LinkOption;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+
+import org.apache.camel.CamelExecutionException;
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.file.GenericFileOperationFailedException;
+import org.apache.camel.component.file.remote.SftpEndpoint;
+import org.apache.camel.component.file.remote.SftpOperations;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.test.infra.ftp.services.embedded.SftpEmbeddedService;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.apache.camel.util.function.ThrowingRunnable;
+import org.apache.sshd.server.session.ServerSession;
+import org.apache.sshd.sftp.common.SftpConstants;
+import org.apache.sshd.sftp.common.SftpException;
+import org.apache.sshd.sftp.server.DirectoryHandle;
+import org.apache.sshd.sftp.server.SftpEventListener;
+import org.apache.sshd.sftp.server.SftpFileSystemAccessor;
+import org.apache.sshd.sftp.server.SftpSubsystemFactory;
+import org.apache.sshd.sftp.server.SftpSubsystemProxy;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIf;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The sftp component checks whether a file exists, and reads its size and
modification time for readLock=changed, with
+ * a single LSTAT request, whatever the number of files in the directory
(CAMEL-26656). It lists the directory only when
+ * the server does not answer that request for the path.
+ */
+@EnabledIf(value =
"org.apache.camel.test.infra.ftp.services.embedded.SftpUtil#hasRequiredAlgorithms('src/test/resources/hostkey.pem')")
+class SftpExistsFileIT extends CamelTestSupport {
+
+ // a reply of the server to a directory listing request holds about 150 of
these entries
+ private static final int FILES = 1000;
+
+ @RegisterExtension
+ CountingSftpEmbeddedService service = new CountingSftpEmbeddedService();
+
+ private Path base;
+
+ @BeforeEach
+ void createTestDirectory() throws IOException {
+ // the root directory of the server is shared by the tests of this
class and kept between runs
+ base =
Files.createTempDirectory(Files.createDirectories(service.getFtpRootDir()),
"test");
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = { false, true })
+ void existsFileIsOneRequestWhateverTheDirectorySize(boolean
fastExistsCheck) throws Exception {
+ createFiles(base.resolve("small"), 10);
+ createFiles(base.resolve("big"), FILES);
+ Files.createDirectories(base.resolve("big").resolve("sub"));
+
+ SftpOperations operations = connect(fastExistsCheck);
+ try {
+ for (String directory : new String[] { "small", "big" }) {
+ assertExists(operations, true, directory + "/file-0005.txt");
+ assertExists(operations, false, directory + "/missing.txt");
+ }
+ assertExists(operations, true, "big/sub");
+ assertExists(operations, false, "nodir/missing.txt");
+ } finally {
+ operations.disconnect();
+ }
+ }
+
+ @ParameterizedTest
+ @CsvSource({
+ "PERMISSION_DENIED, false", "OP_UNSUPPORTED, false", "FAILURE,
false", "BAD_MESSAGE, false",
+ "NO_SUCH_PATH, false",
+ "PERMISSION_DENIED, true", "OP_UNSUPPORTED, true", "FAILURE,
true", "BAD_MESSAGE, true",
+ "NO_SUCH_PATH, true" })
+ void listsDirectoryWhenServerDoesNotAnswerLstat(String status, boolean
fastExistsCheck) throws Exception {
+ createFiles(base.resolve("inbox"), 10);
+ service.onLstat(() -> {
+ switch (status) {
+ case "PERMISSION_DENIED" -> throw new
AccessDeniedException("lstat");
+ case "OP_UNSUPPORTED" -> throw new
UnsupportedOperationException("lstat");
+ // a status that does not say whether the path exists, and one
of a later version of the protocol,
+ // which some servers send
+ case "BAD_MESSAGE" -> throw new
SftpException(SftpConstants.SSH_FX_BAD_MESSAGE, "lstat");
+ case "NO_SUCH_PATH" -> throw new
SftpException(SftpConstants.SSH_FX_NO_SUCH_PATH, "lstat");
+ default -> throw new IOException("lstat");
+ }
+ });
+
+ SftpOperations operations = connect(fastExistsCheck);
+ try {
+ service.reset();
+
assertTrue(operations.existsFile(remotePath("inbox/file-0005.txt")));
+
assertFalse(operations.existsFile(remotePath("inbox/missing.txt")));
+ assertEquals(2, service.requests(SftpConstants.SSH_FXP_LSTAT));
+ // the existing file was found in a listing of its directory, as
before
+ assertTrue(service.requests(SftpConstants.SSH_FXP_OPENDIR) >= 1,
"requests: " + service.requests);
+ } finally {
+ operations.disconnect();
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = { false, true })
+ void listsDirectoryWhenServerAnswersNeitherLstatNorStatOfFile(boolean
fastExistsCheck) throws Exception {
+ createFiles(base.resolve("inbox"), 10);
+ // a server that does not permit LSTAT on a path does not permit STAT
either, while it lists the directory
+ service.refuseAttributesOf(path -> !Files.isDirectory(path));
+
+ SftpOperations operations = connect(fastExistsCheck);
+ try {
+ service.reset();
+
assertTrue(operations.existsFile(remotePath("inbox/file-0005.txt")));
+
assertFalse(operations.existsFile(remotePath("inbox/missing.txt")));
+ assertEquals(2, service.requests(SftpConstants.SSH_FXP_LSTAT));
+ // both were looked up in a listing of their directory
+ assertEquals(2, service.requests(SftpConstants.SSH_FXP_OPENDIR),
"requests: " + service.requests);
+ } finally {
+ operations.disconnect();
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = { false, true })
+ void failsWhenConnectionIsLostDuringLstat(boolean fastExistsCheck) throws
Exception {
+ createFiles(base.resolve("inbox"), 10);
+
+ SftpOperations operations = connect(fastExistsCheck);
+ try {
+ service.reset();
+ // the server drops the connection when it receives the LSTAT
request, before it answers
+ service.onLstat(service::disconnectAllSessions);
+ GenericFileOperationFailedException e =
assertThrows(GenericFileOperationFailedException.class,
+ () ->
operations.existsFile(remotePath("inbox/file-0005.txt")));
+ assertEquals(1, service.requests(SftpConstants.SSH_FXP_LSTAT));
+ // the check fails, it does not answer that the file is missing.
JSch gives the SftpException of this I/O
+ // failure a cause, and none to the SftpException of a status sent
by the server, which tells them apart
+ assertNotNull(e.getCause().getCause(), "cause of: " +
e.getCause());
+ } finally {
+ operations.disconnect();
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = { false, true })
+ void danglingSymbolicLinkExists(boolean fastExistsCheck) throws Exception {
+ Path inbox = Files.createDirectories(base.resolve("inbox"));
+ createSymbolicLink(inbox.resolve("link.txt"), Path.of("missing.txt"));
+
+ SftpOperations operations = connect(fastExistsCheck);
+ try {
+ // the path itself, as in the directory listing: the link exists,
though its target does not
+ assertExists(operations, true, "inbox/link.txt");
+ assertExists(operations, false, "inbox/missing.txt");
+ } finally {
+ operations.disconnect();
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = { false, true })
+ void fileExistFailDoesNotListDirectory(boolean fastExistsCheck) throws
Exception {
+ Path out = base.resolve("out");
+ createFiles(out, FILES);
+ String uri = sftpUri("out") +
"&autoCreate=false&fileExist=Fail&fastExistsCheck=" + fastExistsCheck;
+ // connect before counting
+ template.sendBodyAndHeader(uri, "first", Exchange.FILE_NAME,
"first.txt");
+ service.reset();
+
+ CamelExecutionException e = assertThrows(CamelExecutionException.class,
+ () -> template.sendBodyAndHeader(uri, "replaced",
Exchange.FILE_NAME, "file-0005.txt"));
+ assertInstanceOf(GenericFileOperationFailedException.class,
e.getCause());
+ template.sendBodyAndHeader(uri, "written", Exchange.FILE_NAME,
"new.txt");
+
+ assertEquals("x", Files.readString(out.resolve("file-0005.txt")));
+ assertEquals("written", Files.readString(out.resolve("new.txt")));
+ assertEquals(2, service.requests(SftpConstants.SSH_FXP_LSTAT));
+ assertEquals(0, service.requests(SftpConstants.SSH_FXP_OPENDIR),
"requests: " + service.requests);
+ assertEquals(0, service.entriesListed());
+ }
+
+ @Test
+ void tempFileNameReplacesExistingFileWithoutListingDirectory() throws
Exception {
+ Path out = base.resolve("out");
+ createFiles(out, FILES);
+ String uri = sftpUri("out") +
"&autoCreate=false&tempFileName=${file:onlyname}.inprogress";
+ // connect before counting
+ template.sendBodyAndHeader(uri, "first", Exchange.FILE_NAME,
"first.txt");
+ service.reset();
+
+ template.sendBodyAndHeader(uri, "replaced", Exchange.FILE_NAME,
"file-0005.txt");
+
+ assertEquals("replaced",
Files.readString(out.resolve("file-0005.txt")));
+ assertFalse(Files.exists(out.resolve("file-0005.txt.inprogress")));
+ // the target and the temp file
+ assertEquals(2, service.requests(SftpConstants.SSH_FXP_LSTAT));
+ assertEquals(0, service.requests(SftpConstants.SSH_FXP_OPENDIR),
"requests: " + service.requests);
+ assertEquals(0, service.entriesListed());
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = { false, true })
+ void readLockChangedDoesNotListDirectory(boolean fastExistsCheck) throws
Exception {
+ createFiles(base.resolve("inbox"), FILES);
+
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+
+ consume("inbox",
"&maxMessagesPerPoll=1&readLock=changed&readLockCheckInterval=10&fastExistsCheck="
+ + fastExistsCheck);
+
+ mock.assertIsSatisfied();
+ // at least two checks, with the same size and modification time
+ assertTrue(service.requests(SftpConstants.SSH_FXP_LSTAT) >= 2,
"requests: " + service.requests);
+ // only the poll lists the directory, and stops at the first file
+ assertEquals(1, service.requests(SftpConstants.SSH_FXP_OPENDIR),
"requests: " + service.requests);
+ assertTrue(service.entriesListed() < FILES, "entries listed: " +
service.entriesListed());
+ }
+
+ @Test
+ void readLockChangedWaitsUntilFileStopsChanging() throws Exception {
+ Path inbox = Files.createDirectories(base.resolve("inbox"));
+ Path file = inbox.resolve("growing.txt");
+ Files.writeString(file, "line 0\n");
+ // the file grows before each of the first three checks reads its
attributes
+ AtomicInteger checks = new AtomicInteger();
+ service.onLstat(() -> {
+ int check = checks.incrementAndGet();
+ if (check <= 3) {
+ Files.writeString(file, "line " + check + "\n",
StandardOpenOption.APPEND);
+ }
+ });
+
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("line 0\nline 1\nline 2\nline 3\n");
+
+ consume("inbox", "&readLock=changed&readLockCheckInterval=10");
+
+ mock.assertIsSatisfied();
+ // the last two checks found the same size and modification time
+ assertTrue(checks.get() >= 4, "checks: " + checks.get());
+ }
+
+ private SftpOperations connect(boolean fastExistsCheck) {
+ SftpEndpoint endpoint = context.getEndpoint(sftpUri("") +
"&fastExistsCheck=" + fastExistsCheck, SftpEndpoint.class);
+ SftpOperations operations = (SftpOperations)
endpoint.createRemoteFileOperations();
+ operations.connect(endpoint.getConfiguration(), null);
+ // JSch asks the server for the home directory once, to resolve the
relative paths against it
+ operations.getCurrentDirectory();
+ return operations;
+ }
+
+ private void assertExists(SftpOperations operations, boolean expected,
String name) {
+ service.reset();
+ assertEquals(expected, operations.existsFile(remotePath(name)), name);
+ // a single LSTAT request, and no directory listing
+ assertEquals(1, service.requests(SftpConstants.SSH_FXP_LSTAT), name +
", requests: " + service.requests);
+ assertEquals(1, service.requests(), name + ", requests: " +
service.requests);
+ assertEquals(0, service.entriesListed(), name);
+ }
+
+ private String remotePath(String name) {
+ // relative to the home directory of the server, as the test directory
+ return base + "/" + name;
+ }
+
+ private String sftpUri(String directory) {
+ return "sftp://localhost:"; + service.getPort() + "/" + base + "/" +
directory
+ + "?username=admin&password=admin&knownHostsFile=" +
service.getKnownHostsFile();
+ }
+
+ private void consume(String directory, String query) throws Exception {
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ // autoCreate checks that the directory exists by reading the
first reply of a listing
+ from(sftpUri(directory) +
"&autoCreate=false&initialDelay=0&delay=60000&noop=true" + query)
+ .convertBodyTo(String.class)
+ .to("mock:result");
+ }
+ });
+ }
+
+ private static void createFiles(Path directory, int count) throws
IOException {
+ Files.createDirectories(directory);
+ for (int i = 0; i < count; i++) {
+ Files.writeString(directory.resolve(String.format("file-%04d.txt",
i)), "x");
+ }
+ }
+
+ private static void createSymbolicLink(Path link, Path target) {
+ try {
+ Files.createSymbolicLink(link, target);
+ } catch (FileAlreadyExistsException e) {
+ throw new IllegalStateException(e);
+ } catch (UnsupportedOperationException | FileSystemException e) {
+ // such as Windows when the user may not create symbolic links
+ Assumptions.abort("Symbolic links cannot be created on this
platform: " + e.getMessage());
+ } catch (IOException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ /**
+ * The embedded SFTP server, counting the requests it receives and the
directory entries it sends, and running a
+ * hook before it answers an LSTAT request: a hook that throws makes the
server answer with an error status. It can
+ * also refuse the LSTAT and STAT requests on some paths only.
+ */
+ static class CountingSftpEmbeddedService extends SftpEmbeddedService {
+
+ // by the name of the request type, such as SSH_FXP_LSTAT
+ private final Map<String, AtomicInteger> requests = new
ConcurrentHashMap<>();
+ private final AtomicInteger entriesListed = new AtomicInteger();
+ private volatile ThrowingRunnable<IOException> onLstat;
+ private volatile Predicate<Path> refuseAttributesOf;
+
+ @Override
+ public void setUpServer() throws Exception {
+ super.setUpServer();
+ SftpSubsystemFactory factory = (SftpSubsystemFactory)
sshd.getSubsystemFactories().get(0);
+ factory.setFileSystemAccessor(new SftpFileSystemAccessor() {
+ @Override
+ public LinkOption[] resolveFileAccessLinkOptions(
+ SftpSubsystemProxy subsystem, Path file, int cmd,
String extension, boolean followLinks)
+ throws IOException {
+ Predicate<Path> refused = refuseAttributesOf;
+ if ((cmd == SftpConstants.SSH_FXP_LSTAT || cmd ==
SftpConstants.SSH_FXP_STAT) && refused != null
+ && refused.test(file)) {
+ throw new AccessDeniedException(file.toString());
+ }
+ return
SftpFileSystemAccessor.super.resolveFileAccessLinkOptions(
+ subsystem, file, cmd, extension, followLinks);
+ }
+ });
+ factory.addSftpEventListener(new SftpEventListener() {
+ @Override
+ public void received(ServerSession session, int type, int id)
throws IOException {
+
requests.computeIfAbsent(SftpConstants.getCommandMessageName(type), k -> new
AtomicInteger())
+ .incrementAndGet();
+ ThrowingRunnable<IOException> hook = onLstat;
+ if (type == SftpConstants.SSH_FXP_LSTAT && hook != null) {
+ hook.run();
+ }
+ }
+
+ @Override
+ public void readEntries(
+ ServerSession session, String remoteHandle,
DirectoryHandle localHandle, Map<String, Path> entries) {
+ entriesListed.addAndGet(entries.size());
+ }
+ });
+ }
+
+ void onLstat(ThrowingRunnable<IOException> hook) {
+ this.onLstat = hook;
+ }
+
+ /**
+ * Answers the LSTAT and STAT requests on the paths that match with
PERMISSION_DENIED.
+ */
+ void refuseAttributesOf(Predicate<Path> paths) {
+ this.refuseAttributesOf = paths;
+ }
+
+ void reset() {
+ requests.clear();
+ entriesListed.set(0);
+ }
+
+ int requests(int type) {
+ AtomicInteger count =
requests.get(SftpConstants.getCommandMessageName(type));
+ return count != null ? count.get() : 0;
+ }
+
+ int requests() {
+ return
requests.values().stream().mapToInt(AtomicInteger::get).sum();
+ }
+
+ int entriesListed() {
+ return entriesListed.get();
+ }
+ }
+}
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index dbc8f371bfdd..db311e0e5996 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -163,6 +163,26 @@ Apache MINA SSHD leaves it off by default, so a small
request, such as one that
listing, could wait for the server to acknowledge the previous reply. The
`org.apache.sshd.config.tcp-nodelay` system
property no longer applies to the connections of `mina-sftp`.
+=== camel-ftp - sftp checks whether a file exists with one lstat request
+
+The `sftp` component now checks whether a remote file exists
(`fileExist=Ignore|Fail|Move`, `tempFileName`,
+`tempPrefix`, a producer `doneFileName`), and reads the size and modification
time of the file for `readLock=changed`,
+with a single `lstat` request instead of a listing of the directory. Each
check is one request, whatever the number of
+files in the directory, and `fastExistsCheck` no longer makes a difference for
these checks. The answer can differ from
+before in two cases:
+
+* A dangling symbolic link now exists also with `fastExistsCheck=true`, as it
already did without that option: the
+ link itself is checked, not its target.
+* On a server with a case-insensitive file system, such as one backed by
Windows or macOS, a file whose name differs
+ only in case now exists: before, the names of the listing were compared
case-sensitively. For example, with
+ `fileExist=Fail`, sending `REPORT.TXT` to a directory that holds
`report.txt` now fails, where before it replaced
+ `report.txt`.
+
+When the server answers `lstat` with another status than "no such file", for
example because it does not permit or
+support it on the path, the component lists the directory as before.
`readLock=changed` keeps comparing the size and
+modification time of a symbolic link itself, as the directory listing reported
them. See the
+xref:components::sftp-component.adoc[SFTP component].
+
=== Java
*Breaking change:* Camel 4.24 requires *Java 21* as the minimum runtime
version.
