[
https://issues.apache.org/jira/browse/SSHD-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17435276#comment-17435276
]
Thomas Wolf commented on SSHD-1217:
-----------------------------------
[~roberto.deandrea], thanks for this log. It's interesting.
* First thing I see (and already knew :( ) is that the logging in Apache MINA
sshd is absolutely terrible. Most trace logs just give LoggingUtils as origin,
which is rather useless. It also logs completely irrelevant stuff. Some
improvements there are needed.
* Second thing I see is that the front-end server works as it should.
* Third thing I see is that either your back-end server is slow, or the
network between the two is slow.
But please do some simple log analysis yourself first. Don't just dump tons of
raw logs into my lap. I'm not the unpaid log analyzer of your employer.
Surely a company developing a DMZ Gateway for some data integration platform
has much better tools to do log analysis than me. You surely would have Splunk
or something like it. I just have grep. (OK, Splunk is just a glorified grep.)
I'm way too kind and did take a look all the same:
Do a {{grep -E 'seding|FXP_REALPATH|FXP_READDIR|FXP_NAME|FX_EOF'
sftp-sshd-fe-listing-pr206.log}} on this. (And sorry about the "seding" typo in
the logging output; I'll fix that.) You'll find that the change works as
expected, but:
* You are running with different MAX_READDIR_DATA_SIZE settings. It seems your
back-end server runs with the default of 16kB, while the front-end (the DMZ
Gateway) uses a value about 2-3 times higher? That shouldn't cause problems,
though.
** How do I know? The front-end server makes three SSH_FXP_READDIR calls to
the back-end for each SSH_FXP_READDIR is received from the client, then sends
back >300 entries.
* Directory listing starts at [10/28/21 7:30:16:130 GMT] and ends at [10/28/21
7:31:21:961 GMT]. So we're looking at nearly 66s, as you mentioned.
* Each of these three SSH_FXP_READDIR from the front-end to the back-end
server takes a lot of time. It takes a lot of time between the front-end
sending the SSH_FXP_READDIR to the back-end until the SSH_FXP_NAME reply is
received by your front-end. Look at the pairs of SSH_FXP_READDIR/SSH_FXP_NAME:
they take 31s, 3s, 4s, (reply to client), 4s, 3s, reply, and so on. Don't know
why the first one is so much higher. Let's do some estimates:
** It takes about 3-4s to get 16kb of directory listing from your back-end to
your front-end. (About 150 entries). Let's say 3s.
** 2000 / 150 = 13.333, so 14 calls from front-end to back-end in total
** 13 * 3 + 31 = 70 (first call takes 31s, the remaining 13 take 3s)
** From this ballpark calculation it looks as if _all the time gets lost in
the communication between the two servers, and your back-end server is slow to
respond_.
Therefore, analyze the network setup between your two servers, and analyze why
that back-end server takes so much time to deliver its directory listing.
> Slow performance listing huge number of files on Apache SSHD server
> -------------------------------------------------------------------
>
> Key: SSHD-1217
> URL: https://issues.apache.org/jira/browse/SSHD-1217
> Project: MINA SSHD
> Issue Type: Improvement
> Affects Versions: 2.6.0
> Reporter: Roberto Deandrea
> Assignee: Thomas Wolf
> Priority: Minor
> Attachments: sftp-sshd-fe-listing-pr206.log.zip,
> trace.ssh-frontend-sftplist.finest.log.zip
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Hi Thomas,
> I noted slow performance listing files on the front-end Apache SSHD server in
> the same scenario as https://issues.apache.org/jira/browse/SSHD-1215
> The front-end Apache SSHD server is configured with a Filesystem built upon
> SFTPFileSystemProvider to proxy files to an Apache SSHD back-end server.
>
> In the /inbox folder of the Apache SSHD backend server I have 2000 files.
> The client sftp ls commands take 2 secs on the backend Apache SSHD server,
> instead it takes about 48 secs on the front-end Apache SSHD server.
> For greater number of files in the /inbox folder times are getting worse.
>
> I have full traces of sftp list commands to front-end Apache SSHD server
> that is attached to this jira.[^trace.frontend.sshd.log.zip]
> I looked through the traces on the front-end server and it seems to me that
> for every files in the folder the sftp client on the front-end server creates
> a SSH_MSG_CHANNEL_DATA generating tcp traffic that slow down the performance
> of the list command.
> Obviously this does not happen when a sftp client connects directly to the
> backend Apache SSHD server.
> Can you take a look at traces on the front-end Apache SSHD server ?
> Do you think it's possbile change something to improve performance of list
> files in this situation ?
>
> Thanks in advance
>
> Kind Regards
> Roberto
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
