Hi Josef, Thanks for the reply, and for highlighting the MaxStartups setting in OpenSSH. As the documentation notes, MaxStartups limits the number of unauthenticated connections, so for the purpose of PutSFTP, it is probably more important to have a sufficiently high number allowed.
I can appreciate the challenge of clear flow design with multiple sources channeling to a single PutSFTP Processor. Given the Synology NAS support for 4000 concurrent connections, it sounds like it should be more than capable of handling the number of SFTP connections from NiFi. If you run into additional issues, please pass along the details. Regards, David Handermann On Fri, Feb 17, 2023 at 3:44 PM <[email protected]> wrote: > Hi David > > > > Thanks a lot for answering here in this topic. > > > > A few comments to your reply. > > > > - SSHD config and the “MaxSession” setting -> when we started > troubleshooting we thought as well that this could help to fix our issue. > However based on the sshd_config manpage (see below) this setting is > correlating only to session multiplexing, which I don’t expect that NiFi > does it. That was one of the reasons why I created my mail. When we started > using the SFTP Server increased as well the “MaxStartups” to 100 as > potentially a lot of PutSFTP processors can try to open a connection to the > SFTP server in parallel. > > > *MaxSessions* > > *Specifies the maximum number of open shell, login or subsystem (e.g. > sftp) sessions permitted per network connection. Multiple sessions may be > established by clients that support connection multiplexing. Setting > MaxSessions to 1 will effectively disable session multiplexing, whereas > setting it to 0 will prevent all shell, login and subsystem sessions while > still permitting forwarding. The default is 10.* > > > > - Reason for having 50 PutSFTP instance -> We fetch data from a lot of > different data sources, each of them are processed and the original files > will be store on the SFTP server. The reason why we created not a single > SFTP processor was clarity on the GUI. It would need a lot of local > Input/Output ports and would mess up the canvas. But you are right, less > SFTP processor would reduce the load of the SFTP server as we would use > less total processors to transfer data. However we use a quite powerful > Synology NAS (SA3600) as SFTP server and based on the datasheet it should > be possible to handle 4000 concurrent sessions (sadly not more). > > > > Based on your explanation how NiFi handles SFTP we decided to restart and > upgrade our NAS to the newest firmware. In the last few hours the issue > happened only two times, but we will monitor it. It’s very likely that the > issue is or was related to our SFTP server and not to NiFi. > > > > Cheers Josef > > > > *From: *David Handermann <[email protected]> > *Date: *Thursday, 16 February 2023 at 15:51 > *To: *[email protected] <[email protected]> > *Subject: *Re: NiFi 1.20.0 PutSFTP: SSH Client connection failed -> > Timeout expired > > Hi Josef, > > > > The FetchSFTP Processor implements some connection reuse, but GetSFTP, > ListSFTP, and PutSFTP create a new connection for every invocation of the > processor. I have considered a new approach SFTP components using a > Controller Service with connection pooling, but it still requires some > design work prior to implementation. > > > > Based on the current SFTP processor behavior, it is possible to have > connection timeouts when the SFTP server is not accepting new connections. > Every SFTP server is different, but OpenSSH uses the MaxSessions setting in > sshd_config [1] to limit the number of simultaneous open sessions. > > > > Using the example of 50 PutSFTP processors connecting to the same server, > it is very possible to encounter a connection timeout if NiFi triggers all > of them in rapid succession. > > > > The Connection Timeout property in PutSFTP controls how long the processor > will wait before throwing the ClientConnectException. The number of > Concurrent Tasks for each PutSFTP processor also influences the number of > simultaneous connections. Increasing the Connection Timeout in PutSFTP may > hide the problem, but if the destination SFTP server can handle the load, > it may be helpful to increase the number of maximum sessions. > > > > On the other hand, is there a reason for having 50 separate instances of > PutSFTP sending to the same server? It should be possible to design the > flow and parameterize the destination using FlowFile attributes. Depending > on the number of CPU cores and available threads, having more SFTP > connections results in poor performance, so smaller numbers can be better. > > > > Regards, > > David Handermann > > > > [1] https://linux.die.net/man/5/sshd_config > <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinux.die.net%2Fman%2F5%2Fsshd_config&data=05%7C01%7CJosef.Zahner1%40swisscom.com%7Cca0dea0c3781417181e008db102d598a%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C638121559173781274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CjGVtb6tqsQsNelhECCV3aNvvFwgZwEYEC9gGiEnWwQ%3D&reserved=0> > > > > On Thu, Feb 16, 2023 at 1:33 AM <[email protected]> wrote: > > Hi guys > > > > It was upgrade time again on our side, we just upgraded from 1.19.1 to > 1.20.0. Since 1.20.0 we do see significantly more SSH Connection Timeout > errors on the PutSFTP processor… > > > > PutSFTP Processor ERROR: > > 2023-02-16 07:44:07,905 ERROR [Timer-Driven Process Thread-50] > o.a.nifi.processors.standard.PutSFTP PutSFTP[ > id=12563431-c40a-1af7-b09b-16de27d887b7] Unable to transfer > StandardFlowFileRecord[uuid=a1adadb1-61f9-414f-99e5-aad4331165ef, > claim=StandardContentClaim [resourceClaim=StandardResourceClaim[ > id=1676529847583-196507, container=default, section=923], offset=0, > length=6582249],offset=0,name=myfile.avro.gz,size=6582249] to remote host > nas.local.com > <https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnas.local.com%2F&data=05%7C01%7CJosef.Zahner1%40swisscom.com%7Cca0dea0c3781417181e008db102d598a%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C638121559173781274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8u5wureyR4zVU%2FUdXb6TqTWpY9V890qTb3a1cTe%2BVt4%3D&reserved=0> > due to org.apache.nifi.processors.standard.socket.ClientConnectException: > SSH Client connection failed [nas.local.com > <https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnas.local.com%2F&data=05%7C01%7CJosef.Zahner1%40swisscom.com%7Cca0dea0c3781417181e008db102d598a%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C638121559173781274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8u5wureyR4zVU%2FUdXb6TqTWpY9V890qTb3a1cTe%2BVt4%3D&reserved=0> > :22]: net.schmizz.sshj.transport.TransportException: Timeout expired: > 30000 MILLISECONDS; routing to failure > net.schmizz.sshj.transport.TransportException: Timeout expired: 30000 > MILLISECONDS > > > > I know that David Handermann implemented a fix ( > https://issues.apache.org/jira/browse/NIFI-9989 > <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9989&data=05%7C01%7CJosef.Zahner1%40swisscom.com%7Cca0dea0c3781417181e008db102d598a%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C638121559173781274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lojp%2BvP%2BoGq9DVzZnFsugKh0FdYgdP4prYmtsXsoAZI%3D&reserved=0>) > for SSHJ, but I don’t know if it really is related. May be it’s just a > configuration issue (number of allowed concurrent connections?) on the SFTP > Server side. > > Let’s make an example, let’s say we do have 50 PutSFTP processors to the > same destination and all of them are getting data in an interval of 15min. > Does NiFi keep the SSH connection established for this 50 processors or > will it be closed after each flow has been transferred? If it isn’t closed > after each flow, how can we influence the timeout? I see only the two > timeouts below, which let me assume that it’s closed after each flow… But > may be one of you guys can bring some light into the dark. > > > > [image: Background pattern Description automatically generated with low > confidence] > > > > > > Cheers > > Josef > >
