Re: [paramiko] Multithreading
On 07/01/2010 12:43 PM, james bardin wrote: On Thu, Jul 1, 2010 at 12:06 PM, Nikolaus Rath nikol...@rath.org wrote: You can share it, like you can share anything you want between threads, you need proper locking, as the client only has one channel for communication. Since I can share anything I want if I synchronize access to it myself, my question was meant as can I share it without explicit locking. Generally in python, the only objects you can share without explicit locking are single instances of core data types - basically lists and dicts. As well as third-party modules that have been designed to be threadsafe. Best, -Niko -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko
Re: [paramiko] Multithreading
On 07/01/2010 01:28 PM, james bardin wrote: On Thu, Jul 1, 2010 at 1:00 PM, Nikolaus Rathnikol...@rath.org wrote: Generally in python, the only objects you can share without explicit locking are single instances of core data types - basically lists and dicts. As well as third-party modules that have been designed to be threadsafe. *Modules* can be thread-safe, but what's an example of a module that advertises it's classes as thread-safe? Any class that does that would need to self-lock on all non-atomic operations. It's normally just easier to expect locking to be handled outside of the class. In most cases you are probably right. But I think there are also good cases where locking is better done in the class itself. The class does not need to self-lock all non-atomic operations, only those that actually operate on instance (or global) variables. And even when locking is required, the method is able to lock just the one variable it is working with rather than the entire method. Example: I am working with a class that uploads data to Amazon S3 (basically an online storage service with a simple HTTP API). The class provides methods like put_from_fh(key, fh) and get_to_fh(key, fh) which are designed to be threadsafe. When called, they first compress and encrypt the data, then they briefly obtain a lock to get a HTTP connection from a pool, release the lock and upload the data. The methods have to be multithreaded because they provide a file system backend (and it would be rather annoying if you would have to wait for you 100 MB write into file1 to complete before you can read 10 bytes from file2). There are of course other possible implementations, like giving every thread its own S3 storage instance or managing the storage classes in a pool (instead of the HTTP connections), but I consider those to be less elegant. The S3 storage class has semantics like a dict (only that the amount of stored data is larger and stored elsewhere), so I would consider it quite awkward if I had to bother with locking or pooling when using it. Best, -Nikolaus Btw, I am about to write a class that provides the same functionality over SFTP, thus my initial question. -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko
Re: [paramiko] Multithreading
On 06/30/2010 09:03 AM, Marcin Krol wrote: Nikolaus Rath wrote: Hello, I would like to use an SFTPClient instance concurrently with several threads, but I couldn't find any information about thread safety in the API documentation. - Can I just share the SFTPClient instance between several threads? Why use SFTPClient? Its performance might not be very good plus compatibility issues with some SSH servers might crop up. I know I had compatibility issues even with some pretty standard SSH servers on some platforms (esp. Solaris). I would avoid SFTPClient if I were you. I wrote my own multithreaded SCP class (handling both upload and download) which I can post if you're interested. It's been in use for a while by several users and I think it's pretty well debugged by now. All I need is a Python API for uploading, downloading and renaming files over SSH. I chose SFTPClient since it seemed to be the simplest solution, and I don't remember seeing any warnings about performance or compatibility. Can you tell me what exactly the problem with SFTPClient is? Are there any better options within paramiko? In any case, I am certainly interested in taking a look at your solution. There's quite a number of rather nasty problems you need to deal with: for instance, what if you need to close down the thread in the middle of operation but SFTPClient doesn't allow that? When I'm in the middle of an operation, then I am in the middle of an SFTPClient method. Obviously I can't shut down the thread while the interpreter is not executing my code. This doesn't seem to be an SFTPClient or even multithreading specific problem to me. What if you're shutting down an interpreter and SFTPClient throws an exception which is visible for end user? The interpreter should keep running while at least one thread is alive. It seems to me that if SFTPClient throws an exception, obviously something went wrong and it is a good think to know about it. What if there's no SCP/sftp on the other end (and this does happen from time to time) If the user tries to establish an SFTP connection to a server that does not support SFTP, then things will obviously break. But that's not a bug in the program. I have done quite a lot of work on getting my class to work reasonably under such circumstances: for instance, thread's file sending/downloading methods watch value of thread.abort flag and if it's set to True by an external class, they shut down gracefully. I think you are using multithreading in a different way. Let me guess: you have a main thread that has to stay responsive and therefore delegates time-consuming operations to individual worker threads. These worker threads must shut down when the main threads asks them to do so. In this situation you have to deal with the problems you describe, but they are not specific to SFTPClient and they do not always arise when using multiple threads. For example, my application is much simpler. I have several threads which work independently of each other. There is no main controlling thread. The application terminates when all the threads have finished their work. (I am essentially programming a server with individual threads handling client requests). Obviously, all the normal caveats about multithreading apply: remembering to sleep just in case after releasing locks to prevent starvation I never heard of that. Could you explain in more detail what you mean? Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko
Re: [paramiko] Multithreading
james bardin jbar...@bu.edu writes: On Wed, Jun 30, 2010 at 9:33 AM, Nikolaus Rath nikol...@rath.org wrote: Nikolaus Rath wrote: Hello, I would like to use an SFTPClient instance concurrently with several threads, but I couldn't find any information about thread safety in the API documentation. - Can I just share the SFTPClient instance between several threads? Yes. Though you may have a use case, there's no performance benefit, as the client can only handle one operation at a time. I am mostly concerned about reducing the network latency. Suppose I want to create 100 1-bit files, then I hope that it is going to be faster to send 100 requests at once from 100 threads rather than having one thread that works through the files sequentially. That should still work even with a single threaded client, right? Why use SFTPClient? Its performance might not be very good plus compatibility issues with some SSH servers might crop up. I know I had compatibility issues even with some pretty standard SSH servers on some platforms (esp. Solaris). There are throughput performance issues with older ssh servers that don't support prefetch and pipelining. Ah, so the problem is with the server and not with the SFTPClient class? My servers are given and only speak SFTP, so I am to live with them in any case. Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko
Re: [paramiko] Multithreading
Hi, Really no one around who knows anything about this? -Nikolaus Nikolaus Rath nikol...@rath.org writes: Hello, I would like to use an SFTPClient instance concurrently with several threads, but I couldn't find any information about thread safety in the API documentation. - Can I just share the SFTPClient instance between several threads? - Or can I share the SSHClient object, but each thread needs its own SFTPClient? - Or do I need a separate SSHClient for each thread? - Or is there another layer in between that I can share between threads? - Or do I have to avoid multithreading at all when using paramiko? Thanks! -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko
Re: [paramiko] Handling the background thread
james bardin jbar...@bu.edu writes: On Sat, Mar 13, 2010 at 7:08 PM, Nikolaus Rath nikol...@rath.org wrote: Hello, Is there a way to convince Paramiko to start its background thread as a daemon thread, so that it does not prevent the interpreter from terminating? Are you running the latest version? The transport threads have been set to daemon for a while now, because python 2.6 can't cleanup the threads at exit. I'm running 1.7.4-0.1 (Ubuntu Karmic). Do you know in which version this changed? Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko
[paramiko] Handling the background thread
Hello, Is there a way to convince Paramiko to start its background thread as a daemon thread, so that it does not prevent the interpreter from terminating? I am facing the problem that my program does not terminate when there is an uncaught exception, because the Paramiko background thread is still running. The only alternative seems to be to wrap the entire program into a try .. finally clause. However, this is rather inelegant because I'm using Paramiko as only one of several possible network backends and I would like to make the frontend independent of the used backend. Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ paramiko mailing list paramiko@lag.net http://www.lag.net/cgi-bin/mailman/listinfo/paramiko
