tomaswolf opened a new pull request #206:
URL: https://github.com/apache/mina-sshd/pull/206
An Apache MINA sshd SFTP server configured to use an SftpFileSystem
pointing to yet another SFTP server would serve directory listings
only very slowly.
This was caused by the SFTP server implementation using Java FileSystem
abstractions itself for listing directories. Using java.nio.file.Files,
it ended up doing essentially the following when receiving a
SSH_FXP_READDIR command:
```
try(DirectoryStream<Path> list = Files.newDirectoryStream(dir)) {
Map<Path, BasicFileAttributes> toSend = new HashMap<>();
list.iterator().forEach(p ->
toSend.put(p, Files.readAttributes(p, BasicFileAttributes.class))
);
replyToClient(SSH_FXP_NAME, toSend);
};
```
(Not literally. This omits a lot of special handling for empty
directories, not sending too large messages back to the client, and
so on. But ultimately it boiled down to the above.)
That is fine when the server-side file system from which files are
served is local on the server. But when that file system itself is
a remote file system, `newDirectoryStream` is a remote call sending
SSH_FXP_OPENDIR and SSH_FXP_READDIR to the upstream server, and
additionally each of these `readAttributes()` calls is yet another
remote call sending SSH_FXP_LSTAT. This slows down getting the
directory listing to the client tremendously.
The most annoying part of all this is that the SSH_FXP_READDIR to the
upstream SFTP server in `newDirectoryStream` _already returned all the
attributes_, but this is lost because the Java NIO File abstractions
have no real support for getting a directory listing including
attributes in one fell swoop. (No, using a FileVisitor and
Files.walkFileTree with depth 1 wouldn't help.)
So detect this case in the server's SftpSubsystem, and bypass the Java
FileSystem abstraction if the file system is itself an SftpFileSystem.
Simply issue SSH_FXP_READDIR requests and forward the whole reply, which
includes file names and attributes, directly to the client. For reading
a directory containing 2000 files, this eliminates 10040 SSH_FXP_LSTAT
calls; only the directory itself is stat'ed.
A corollary to this is that clients in general should avoid listing
directories on an SftpFileSystem via java.nio.file.Files _if they also
need the file attributes_. Instead do
```
SftpFileSystem fs = ...;
try (SftpClient client = fs.getClient();
CloseableHandle dir = client.openDir(remDirPath)) {
for (SftpClient.DirEntry entry : client.listDir(dir)) {
Path path = dir.getFile().resolve(entry.getFilename());
SftpClient.Attributes = entry.getAttributes();
// Do whatever you need with it
}
}
```
Implementation note: another idea is to cache the attributes read in
SSH_FXP_READDIR in `newDirectoryStream` on the Path objects returned,
and make `readAttributes` return these cached attributes while the
stream is not closed yet. Technically, this is doable; it'd be a hack
similar to what Java's own UnixFileSystem does (and FileVisitor takes
advantage of it). However, it would break some edge cases, like a
client writing to one of the files during the directory stream
iteration, and then reading the attributes again and expecting the
new attributes but still getting the cached ones.
Side note: the SFTP protocol has no command to get _only_ a listing
of names. SSH_FXP_READDIR _always_ returns both names and attributes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
