Hi Robers On Tue, 22 Sep 2020 at 16:43, Robert DiFalco <robert.difa...@gmail.com> wrote:
> Hey folks, I've cobbled together an SFTP client based on bits and pieces > I've found around the web. The issue is that it appears to be almost one > shot. I will need to send many files (the number not known ahead of time). > It's not clear to me when the connection is closed or how many factories > I'm creating. All the code I've grabbed looks like it's creating a new > factory for every SFTP file I send. Here's some of the code I have. It's > fairly straight forward in that it creates a directory if it doesn't exist > and then writes a file. > > @attr.s(frozen=True) > class FileInfo(object): > """ > Class that tells SFTP details about the file to send. > """ > directory = attr.ib(converter=str) # type: str > name = attr.ib(converter=str) # type: str > data = attr.ib() # type: str > chunk_size = attr.ib(converter=int, default=CHUNK_SIZE) # type: int > > def to_path(self): > """ > Turns the folder and file name into a file path. > """ > return self.directory + "/" + self.name > > > @attr.s(frozen=True) > class SFTPClientOptions(object): > """ > Client options for sending SFTP files. > > :param host: the host of the SFTP server > :param port: the port ofo the SFTP server > :param fingerprint: the expected fingerprint of the host > :param user: the user to login as > :param identity: the identity file, optional and like the "-i" command > line option > :param password: an optional password > """ > host = attr.ib(converter=str) # type: str > port = attr.ib(converter=int) # type: int > fingerprint = attr.ib(converter=str) # type: str > user = attr.ib(converter=str) # type: str > identity = attr.ib(converter=optional(str), default=None) # type: > Optional[str] > password = attr.ib(converter=optional(str), default=None) # type: > Optional[str] > > > @inlineCallbacks > def sftp_send(client_options, file_info): > # type: (SFTPClientOptions, FileInfo)->Deferred > """ > Primary function to send an file over SFTP. You can send a password, > identity, or both. > :param client_options: the client connection options > :param file_info: contains the file info to write > :return: A deferred that signals "OK" if successful. > """ > options = ClientOptions() > options["host"] = client_options.host > options["port"] = client_options.port > options["password"] = client_options.password > options["fingerprint"] = client_options.fingerprint > > if client_options.identity: > options.identitys = [client_options.identity] > > conn = SFTPConnection() > auth = SFTPUserAuthClient(client_options.user, options, conn) > yield connect(client_options.host, client_options.port, options, > _verify_host_key, auth) > > sftpClient = yield conn.getSftpClientDeferred() > yield _send_file(sftpClient, file_info) > > returnValue("OK") > > > def _verify_host_key(transport, host, pubKey, fingerprint): > """ > Verify a host's key. Based on what is specified in options. > > @param host: Due to a bug in L{SSHClientTransport.verifyHostKey}, this is > always the dotted-quad IP address of the host being connected to. > @type host: L{str} > > @param transport: the client transport which is attempting to connect to > the given host. > @type transport: L{SSHClientTransport} > > @param fingerprint: the fingerprint of the given public key, in > xx:xx:xx:... format. > > @param pubKey: The public key of the server being connected to. > @type pubKey: L{str} > > @return: a L{Deferred} which is success or error > """ > expected = transport.factory.options.get("fingerprint", "no_fingerprint") > if fingerprint == expected: > return succeed(1) > > log.error( > "SSH Host Key fingerprint of ({fp}) does not match the expected value > of ({expected}).", > fp=fingerprint, expected=expected) > > return fail(ConchError("Host fingerprint is unexpected.")) > > > class SFTPSession(SSHChannel): > """ > Creates an SFTP session. > """ > name = "session" > > @inlineCallbacks > def channelOpen(self, whatever): > """ > Called when the channel is opened. "whatever" is any data that the > other side sent us when opening the channel. > > @type whatever: L{bytes} > """ > yield self.conn.sendRequest(self, "subsystem", NS("sftp"), > wantReply=True) > > client = FileTransferClient() > client.makeConnection(self) > self.dataReceived = client.dataReceived > self.conn.notifyClientIsReady(client) > > > class SFTPConnection(SSHConnection): > def __init__(self): > """ > Adds a deferred here so client can add a callback when the SFTP > client is ready. > """ > SSHConnection.__init__(self) > self._sftpClient = Deferred() > > def serviceStarted(self): > """ > Opens an SFTP session when the SSH connection has been started. > """ > self.openChannel(SFTPSession()) > > def notifyClientIsReady(self, client): > """ > Trigger callbacks associated with our SFTP client deferred. It's > ready! > """ > self._sftpClient.callback(client) > > def getSftpClientDeferred(self): > return self._sftpClient > > > class SFTPUserAuthClient(SSHUserAuthClient): > """ > Twisted Conch doesn't have a way of getting a password. By default it > gets it from stdin. This allows it > to be retrieved from options instead. > """ > def getPassword(self, prompt = None): > """ > Get the password from the client options, is specified. > """ > if "password" in self.options: > return succeed(self.options["password"]) > > return SSHUserAuthClient.getPassword(self, prompt) > > > @inlineCallbacks > def _send_file(client, file_info): > # type: (FileTransferClient, FileInfo) -> Deferred > """ > Creates a directory if required and then creates the file. > :param client: the SFTP client to use > :param file_info: contains file name, directory, and data > """ > try: > yield client.makeDirectory(file_info.directory, {}) > > except SFTPError as e: > # In testing on various system, either a 4 or an 11 will indicate the > directory > # already exist. We are fine with that and want to continue if it > does. If we misinterpreted > # error code here we are probably still ok since we will just get the > more systemic error > # again on the next call to openFile. > if e.code != 4 and e.code != 11: > raise e > > f = yield client.openFile(file_info.to_path(), FXF_WRITE | FXF_CREAT | > FXF_TRUNC, {}) > > try: > yield _write_chunks(f, file_info.data, file_info.chunk_size) > > finally: > yield f.close() > > > @inlineCallbacks > def _write_chunks(f, data, chunk_size): > # type: (ClientFile, str, int) -> Deferred > """ > Convenience function to write data in chunks > > :param f: the file to write to > :param data: the data to write > :param chunk_size: the chunk size > """ > for offset in range(0, len(data), chunk_size): > chunk = data[offset: offset + chunk_size] > yield f.writeChunk(offset, chunk) > > > It gets called like this: > > return sftp.sftp_send( > client_options=SFTPClientOptions( > host=self.options.host, > port=self.options.port, > user=self.options.user, > fingerprint=self.options.fingerprint, > identity=getattr(self.options, "identity", None), > password=self._getPassword()), > file_info=sftp.FileInfo( > directory=self.options.directory, > name=fileName, > data=data, > chunk_size=getattr(self.options, "chunkSize", sftp.CHUNK_SIZE))) > > But I supposed I'd like to see something more like this: > > sftpClient = self.getSftpClient( > client_options=SFTPClientOptions( > host=self.options.host, > port=self.options.port, > user=self.options.user, > fingerprint=self.options.fingerprint, > identity=getattr(self.options, "identity", None), > password=self._getPassword())) > > return sftpClient.send( > file_info=sftp.FileInfo( > directory=self.options.directory, > name=fileName, > data=data, > chunk_size=getattr(self.options, "chunkSize", sftp.CHUNK_SIZE))) > > Where sftpClient reuses the existing SSH connection if it is active > (rather than logging in each time). But maybe the sftp service doesn't > multiplex so I have to create a new SSHClientFactory every time I want to > send a distinct file? > > Sorry for all the questions, new to twisted and a bit confused. Thanks! > > Robert > > It would help to have the full code...maybe a gist or repo. I am not sure what `connect` from `yield connect(client_options.host, client_options.port, options, _verify_host_key, auth)` is. You will need to understand the low-level Twisted connection API and implement a reconnecting factory. When a new client-side connection is made, Twisted will use a factory to create the protocol/code used to handle that connection, You will then need to hook into the connectionLost method and do an auto-connection if the connection is lost (when you were not expecting it). --------- For my project, I am doing in this way: I have my own subclass of FileTransferClient which overwrites the default FileTransferClient,connectionLost method. With that, I am notified when the SFTP subsystem was closed and I can then trigger a new connection ------------- If you want to reuse an SFTP session for multiple operations just reuse the `sftpClient` instance that you got to trigger multiple operations sftpClient = yield conn.getSftpClientDeferred() for file_info in list_of_files_to_send: yield _send_file(sftpClient, file_info) ---------- Hope it helps -- Adi Roiban
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python