Re: [paramiko] Multithreading
On 06/30/2010 09:03 AM, Marcin Krol wrote: Nikolaus Rath wrote: Hello, I would like to use an SFTPClient instance concurrently with several threads, but I couldn't find any information about thread safety in the API documentation. - Can I just share the SFTPClient instance between several threads? Why use SFTPClient? Its performance might not be very good plus compatibility issues with some SSH servers might crop up. I know I had compatibility issues even with some pretty standard SSH servers on some platforms (esp. Solaris). I would avoid SFTPClient if I were you. I wrote my own multithreaded SCP class (handling both upload and download) which I can post if you're interested. It's been in use for a while by several users and I think it's pretty well debugged by now. All I need is a Python API for uploading, downloading and renaming files over SSH. I chose SFTPClient since it seemed to be the simplest solution, and I don't remember seeing any warnings about performance or compatibility. Can you tell me what exactly the problem with SFTPClient is? Are there any better options within paramiko? In any case, I am certainly interested in taking a look at your solution. There's quite a number of rather nasty problems you need to deal with: for instance, what if you need to close down the thread in the middle of operation but SFTPClient doesn't allow that? When I'm in the middle of an operation, then I am in the middle of an SFTPClient method. Obviously I can't shut down the thread while the interpreter is not executing my code. This doesn't seem to be an SFTPClient or even multithreading specific problem to me. What if you're shutting down an interpreter and SFTPClient throws an exception which is visible for end user? The interpreter should keep running while at least one thread is alive. It seems to me that if SFTPClient throws an exception, obviously something went wrong and it is a good think to know about it. What if there's no SCP/sftp on the other end (and this does happen from time to time) If the user tries to establish an SFTP connection to a server that does not support SFTP, then things will obviously break. But that's not a bug in the program. I have done quite a lot of work on getting my class to work reasonably under such circumstances: for instance, thread's file sending/downloading methods watch value of thread.abort flag and if it's set to True by an external class, they shut down gracefully. I think you are using multithreading in a different way. Let me guess: you have a main thread that has to stay responsive and therefore delegates time-consuming operations to individual worker threads. These worker threads must shut down when the main threads asks them to do so. In this situation you have to deal with the problems you describe, but they are not specific to SFTPClient and they do not always arise when using multiple threads. For example, my application is much simpler. I have several threads which work independently of each other. There is no main controlling thread. The application terminates when all the threads have finished their work. (I am essentially programming a server with individual threads handling client requests). Obviously, all the normal caveats about multithreading apply: remembering to sleep just in case after releasing locks to prevent starvation I never heard of that. Could you explain in more detail what you mean? Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko
Re: [paramiko] Multithreading
Nikolaus Rath wrote:
All I need is a Python API for uploading, downloading and renaming files
over SSH. I chose SFTPClient since it seemed to be the simplest
solution, and I don't remember seeing any warnings about performance or
compatibility.
I don't know about paramiko implementation of SFTPClient, but I tried
using SFTP in my environment (lots of diverse operating systems) and it
didn't work well really: sometimes it worked, sometimes it didn't, so I
moved to SCP and haven't looked back. I didn't really investigate those
problems with SFTP, just got discouraged by them.
Can you tell me what exactly the problem with SFTPClient
is? Are there any better options within paramiko?
Simple SCP.
In any case, I am
certainly interested in taking a look at your solution.
Here:
class SSHThread(threading.Thread):
def __init__(self, hostip, user, passwd, sshprivkey, localpath,
remotepath, action):
threading.Thread.__init__(self)
self.ip = hostip
self.username = user
self.passw = passwd
self.sshprivkey = sshprivkey
self.localpath = localpath
self.remotepath = remotepath
self.action = action
self.port = 22
self.finished = False
self.conerror = ''
self.confailed = False
self.abort = False
self.trans = None
self.sock = None
self.sentbytes = 0
self.socket_timeout = 30
self.result = ''
def run(self):
self.ssh_connect_for_scp()
if self.action == 'download' and not self.confailed:
self.download()
elif self.action == 'upload' and not self.confailed:
self.sendfilesrecursive()
elif self.action == 'listdir' and not self.confailed:
self.listdir()
try:
self.trans.close()
except AttributeError:
pass
self.finished = True
def _get_transport(self):
sock = socket.create_connection((self.ip,self.port),
self.socket_timeout)
trans = paramiko.Transport(sock)
ciphers = trans.get_security_options()._get_ciphers()
if 'blowfish-cbc' in ciphers:
ciphers = ('blowfish-cbc',) + ciphers
if 'arcfour128' in ciphers:
ciphers = ('arcfour128',) + ciphers
trans.get_security_options().ciphers = ciphers
self.sock = sock
self.trans = trans
self.trans.set_log_channel('adserver')
def printandclose(self):
print \ntrying to close transport.sock
#self.trans.close()
self.trans.sock.close()
self.flag = True
def ssh_connect_for_scp(self):
loginsuccess = False
try:
self._get_transport()
except Exception, e:
self.conerror = str(e)
self.confailed = True
self.finished = True
return
# password
if self.passw:
try:
#timer = threading.Timer(4, self.printandclose)
#timer.start()
self.trans.connect(hostkey=None,
username=self.username, password=self.passw, pkey=None)
#self.trans.start_client()
#timer.cancel()
loginsuccess = True
except Exception, e:
self.conerror = str(e)
# key file
pkey = None
if not loginsuccess and self.sshprivkey:
try:
pkey =
paramiko.RSAKey.from_private_key_file(self.sshprivkey)
except paramiko.SSHException:
try:
pkey =
paramiko.DSSKey.from_private_key_file(self.sshprivkey)
except paramiko.SSHException:
pass
except IOError, e:
self.conerror = str(e)
self.finished = True
self.confailed = True
return
if pkey:
try:
#self.conobj.connect(self.ip,
username=self.username, key_filename=self.sshprivkey, port=self.port,
timeout=opts.timeout)
self._get_transport()
self.trans.connect(hostkey=None,
username=self.username, pkey = pkey)
loginsuccess = True
self.conerror = ''
except Exception, e:
self.conerror = str(e)
# agent
#if not loginsuccess:
#agent = paramiko.Agent()
#ak = agent.get_keys()
#for key in ak:
#try:
#self._get_transport()
#self.trans.connect(hostkey=None,
username=self.username, pkey = key )
#loginsuccess = True
#self.conerror = ''
#break
#except Exception, e:
#self.conerror = str(e)
if not loginsuccess:
self.confailed = True
self.conobj = None
self.finished =
Re: [paramiko] Multithreading
On Wed, Jun 30, 2010 at 11:13 AM, Marcin Krol mrk...@gmail.com wrote: Obviously, all the normal caveats about multithreading apply: remembering to sleep just in case after releasing locks to prevent starvation I never heard of that. Could you explain in more detail what you mean? http://linuxgazette.net/107/pai.html Caveat: I don't know if this has been improved in Python thread handling since that article has been written, but I add a bit of sleeping after lock release anyway just to be safe. That's a poor example of python coding. It's highlighting a GIL issue, but if you have that sort of contention, you shouldn't be using multiple threads. The author may have encountered a problem, but he didn't fully understand it, or his own solution; which is why adding a magic sleep seems to work. Note that you will get slightly different results on a multicore system. The main loop is a busy loop! It's hogging the GIL itself, and pegging a cpu core at %100. Because the threads only do work with a lock, there is no time for the GIL to switch threads., The sleep() simply allows a few cycles for the GIL to be released. -jim ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko
Re: [paramiko] Multithreading
james bardin jbar...@bu.edu writes: On Wed, Jun 30, 2010 at 9:33 AM, Nikolaus Rath nikol...@rath.org wrote: Nikolaus Rath wrote: Hello, I would like to use an SFTPClient instance concurrently with several threads, but I couldn't find any information about thread safety in the API documentation. - Can I just share the SFTPClient instance between several threads? Yes. Though you may have a use case, there's no performance benefit, as the client can only handle one operation at a time. I am mostly concerned about reducing the network latency. Suppose I want to create 100 1-bit files, then I hope that it is going to be faster to send 100 requests at once from 100 threads rather than having one thread that works through the files sequentially. That should still work even with a single threaded client, right? Why use SFTPClient? Its performance might not be very good plus compatibility issues with some SSH servers might crop up. I know I had compatibility issues even with some pretty standard SSH servers on some platforms (esp. Solaris). There are throughput performance issues with older ssh servers that don't support prefetch and pipelining. Ah, so the problem is with the server and not with the SFTPClient class? My servers are given and only speak SFTP, so I am to live with them in any case. Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko
