Re: simple and fast platform independent IPC
News123 news...@free.fr writes: Perhaps I'll stick initially with xmlrpc, as it is quite simple, though a little heavy. I just have to see how to deal with servers which are not up, no more up, being restarted. Something wrong wtih nagios? -- http://mail.python.org/mailman/listinfo/python-list
simple and fast platform independent IPC
Hi, I wondered what IPC library might be best simplest for following task? I'm having a few python scripts all running on the same host (linux or win), which are started manually in random order. (no common parent process) Each process might be identified by an integer (1,2,3) or by a symbolic name ( 'dad' , 'mom' , 'dog' ) these scripts want to send short messages to each other ( mostly integers, max a few bytes, short string), which would be enqueued in message queues of the receiving process. example: 'dad' wants to tell 'mom': 'cook' 'dog' wants to tell 'dad' : 'feedme' 'mom' wants to tell 'dad' : 'cookyourself' the receiver dos not necesarily have to know who sent the message a message shall be dropped silently if the receiving process is not running a message shall be reliably delivered if the receiving process is up xmlrpc seems to be a little heavy for such tasks. signals don't allow to exchange data a shared memory message queue would probably a good solution, but python's Multiprocessing.Queue seems to require a common parent process thanks a lot for any ideas / suggestions N N -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
En Wed, 03 Feb 2010 05:32:58 -0300, News123 news...@free.fr escribió: I'm having a few python scripts all running on the same host (linux or win), which are started manually in random order. (no common parent process) Each process might be identified by an integer (1,2,3) or by a symbolic name ( 'dad' , 'mom' , 'dog' ) these scripts want to send short messages to each other ( mostly integers, max a few bytes, short string), which would be enqueued in message queues of the receiving process. example: 'dad' wants to tell 'mom': 'cook' 'dog' wants to tell 'dad' : 'feedme' 'mom' wants to tell 'dad' : 'cookyourself' Try using a named pipe between each pair of processes (os.mkfifo + open on Linux, open(r\\.\pipe\desired_name, ...) on Windows) -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
On Feb 3, 8:32 am, News123 news...@free.fr wrote: Hi, I wondered what IPC library might be best simplest for following task? I'm having a few python scripts all running on the same host (linux or win), which are started manually in random order. (no common parent process) Each process might be identified by an integer (1,2,3) or by a symbolic name ( 'dad' , 'mom' , 'dog' ) these scripts want to send short messages to each other ( mostly integers, max a few bytes, short string), which would be enqueued in message queues of the receiving process. example: 'dad' wants to tell 'mom': 'cook' 'dog' wants to tell 'dad' : 'feedme' 'mom' wants to tell 'dad' : 'cookyourself' the receiver dos not necesarily have to know who sent the message a message shall be dropped silently if the receiving process is not running a message shall be reliably delivered if the receiving process is up xmlrpc seems to be a little heavy for such tasks. signals don't allow to exchange data a shared memory message queue would probably a good solution, but python's Multiprocessing.Queue seems to require a common parent process thanks a lot for any ideas / suggestions N N Gabriel's suggestion is very good; if you need something which is a little more like RPC but still quite lightweight, consider Pyro (http://pyro.sourceforge.net/) Regards, Vinay Sajip -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
On 3 feb, 09:34, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: On Feb 3, 8:32 am, News123 news...@free.fr wrote: Hi, I wondered what IPC library might be best simplest for following task? I'm having a few python scripts all running on the same host (linux or win), which are started manually in random order. (no common parent process) Each process might be identified by an integer (1,2,3) or by a symbolic name ( 'dad' , 'mom' , 'dog' ) these scripts want to send short messages to each other ( mostly integers, max a few bytes, short string), which would be enqueued in message queues of the receiving process. example: 'dad' wants to tell 'mom': 'cook' 'dog' wants to tell 'dad' : 'feedme' 'mom' wants to tell 'dad' : 'cookyourself' the receiver dos not necesarily have to know who sent the message a message shall be dropped silently if the receiving process is not running a message shall be reliably delivered if the receiving process is up xmlrpc seems to be a little heavy for such tasks. signals don't allow to exchange data a shared memory message queue would probably a good solution, but python's Multiprocessing.Queue seems to require a common parent process thanks a lot for any ideas / suggestions N N Gabriel's suggestion is very good; if you need something which is a little more like RPC but still quite lightweight, consider Pyro (http://pyro.sourceforge.net/) Regards, Vinay Sajip I've read that Pyro is not safe. Anyway, you have in mind that respect to speed: shared memory named pipes Unix domain socket TCP socket I don't sure about if the message queues would be faster that Unix domain sockets Another thing. Using shared memory would be as to use a single thread but using message queues would be as multiple-threading. -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
Hi Gabriel, I'll look at it. I wasn't aware about named pipes for windows. bye N Gabriel Genellina wrote: En Wed, 03 Feb 2010 05:32:58 -0300, News123 news...@free.fr escribió: I'm having a few python scripts all running on the same host (linux or win), which are started manually in random order. (no common parent process) Each process might be identified by an integer (1,2,3) or by a symbolic name ( 'dad' , 'mom' , 'dog' ) these scripts want to send short messages to each other ( mostly integers, max a few bytes, short string), which would be enqueued in message queues of the receiving process. example: 'dad' wants to tell 'mom': 'cook' 'dog' wants to tell 'dad' : 'feedme' 'mom' wants to tell 'dad' : 'cookyourself' Try using a named pipe between each pair of processes (os.mkfifo + open on Linux, open(r\\.\pipe\desired_name, ...) on Windows) -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
[News123news...@free.fr] I wondered what IPC library might be best simplest for following task? ... xmlrpc seems to be a little heavy for such tasks. signals don't allow to exchange data a shared memory message queue would probably a good solution, but python's Multiprocessing.Queue seems to require a common parent process [Vinay Sajip] Gabriel's suggestion is very good; if you need something which is a little more like RPC but still quite lightweight, consider Pyro (http://pyro.sourceforge.net/) [pelok...@gmail.com] I've read that Pyro is not safe. That's a fairly broad thing to say. I've read lots of things. What does is not safe mean, in any case? I assume you've got a valid concern in mind which is worth passing on to a would-be user, but what exactly is it? FWIW I've used Pyro on and off over the years without any problems. Certainly my computer's never blown up as a result of using it. Obviously Pyro is Python-only so interaction with non-Python code would be problematic. But the OP only mentions Python scripts so hopefully that wouldn't be an issue... Anyway, you have in mind that respect to speed: shared memory named pipes Unix domain socket TCP socket True, but the OP didn't mention speed; rather simplicity. Not saying it isn't a consideration but premature optimisation and all that... Another thing. Using shared memory would be as to use a single thread but using message queues would be as multiple-threading. And therefore...? I think you need to make your points more clearly. TJG -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
On 03.02.2010 09:32, News123 wrote: Hi, I wondered what IPC library might be best simplest for following task? Consider using Thrift (http://incubator.apache.org/thrift/). It is multiplatform multilanguage RPC and IPC solution. I implemented it in couple of my projects and it works seamlessly. -- www.vikendi.net -/- www.supergrupa.com -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
On 3 feb, 10:54, Tim Golden m...@timgolden.me.uk wrote: [News123news...@free.fr] I wondered what IPC library might be best simplest for following task? ... xmlrpc seems to be a little heavy for such tasks. signals don't allow to exchange data a shared memory message queue would probably a good solution, but python's Multiprocessing.Queue seems to require a common parent process [Vinay Sajip] Gabriel's suggestion is very good; if you need something which is a little more like RPC but still quite lightweight, consider Pyro (http://pyro.sourceforge.net/) [pelok...@gmail.com] I've read that Pyro is not safe. That's a fairly broad thing to say. I've read lots of things. What does is not safe mean, in any case? I assume you've got a valid concern in mind which is worth passing on to a would-be user, but what exactly is it? FWIW I've used Pyro on and off over the years without any problems. Certainly my computer's never blown up as a result of using it. From its own page: Pyro has never been truly designed to provide a secure communication mechanism, nor has it had a security review or -test by a security expert. http://pyro.sourceforge.net/features.html Obviously Pyro is Python-only so interaction with non-Python code would be problematic. But the OP only mentions Python scripts so hopefully that wouldn't be an issue... -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
News123 news...@free.fr writes: I'm having a few python scripts all running on the same host (linux or win), which are started manually in random order. (no common parent process) Each process might be identified by an integer (1,2,3) or by a symbolic name ( 'dad' , 'mom' , 'dog' ) If they are running on the same host with no untrusted local users, you can use unix-domain sockets instead of TCP sockets and then the server should be unreachable from the internet. Then you're less exposed to possible security issues with libraries like Pyro. Personally I've just used SocketServer/SimpleHTTPServer on the listening side and simplejson for serialization, but that was for low-rent, low-performance applications. If you want something faster, zeromq (www.zeromq.org) looks interesting. -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
On 2/3/2010 6:31 AM, Joan Miller wrote: I've read that Pyro is not safe. That's a fairly broad thing to say. I've read lots of things. What does is not safe mean, in any case? I assume you've got a valid concern in mind which is worth passing on to a would-be user, but what exactly is it? FWIW I've used Pyro on and off over the years without any problems. Certainly my computer's never blown up as a result of using it. From its own page: Pyro has never been truly designed to provide a secure communication mechanism, nor has it had a security review or -test by a security expert. http://pyro.sourceforge.net/features.html For communication between processes on one machine, that hardly seems to be a worry. If it were, I would expect that sending encrypted strings would substantially improve things. That aside, I would wonder whether you could use a master process with a gui to haphazardly launch subprocess, so as to avail oneself of multiprocessing.Queue. Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
Tim Golden wrote: Anyway, you have in mind that respect to speed: shared memory named pipes Unix domain socket TCP socket True, but the OP didn't mention speed; rather simplicity. Not saying it isn't a consideration but premature optimisation and all that... Yes true. I'm looking for something simple, platform agnostic. Lateron I can always improve performance. Perhaps I'll stick initially with xmlrpc, as it is quite simple, though a little heavy. I just have to see how to deal with servers which are not up, no more up, being restarted. N -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
Hi Terry, Terry Reedy wrote: That aside, I would wonder whether you could use a master process with a gui to haphazardly launch subprocess, so as to avail oneself of multiprocessing.Queue. T This is also an option I'm looking. insted of the python scripts, thet users would normally click at I had to implement small sripts, that just communicate with the master process, which would then launch the application. -- http://mail.python.org/mailman/listinfo/python-list
Re: simple and fast platform independent IPC
On Feb 3, 3:32 am, News123 news...@free.fr wrote: Hi, I wondered what IPC library might be best simplest for following task? I'm having a few python scripts all running on the same host (linux or win), which are started manually in random order. (no common parent process) Each process might be identified by an integer (1,2,3) or by a symbolic name ( 'dad' , 'mom' , 'dog' ) these scripts want to send short messages to each other ( mostly integers, max a few bytes, short string), which would be enqueued in message queues of the receiving process. example: 'dad' wants to tell 'mom': 'cook' 'dog' wants to tell 'dad' : 'feedme' 'mom' wants to tell 'dad' : 'cookyourself' the receiver dos not necesarily have to know who sent the message a message shall be dropped silently if the receiving process is not running a message shall be reliably delivered if the receiving process is up xmlrpc seems to be a little heavy for such tasks. signals don't allow to exchange data a shared memory message queue would probably a good solution, but python's Multiprocessing.Queue seems to require a common parent process thanks a lot for any ideas / suggestions N N Have a look at the SIMPL toolkit (http://www.icanprogram.com/simpl or http://www.icanprogram.com/06py/main.html). It should do everything you describe at least on the Linux end of the spectrum. Under the hood SIMPL uses a shared memory scheme for local message exchange. SIMPL would be especially effective if you want to move your modules at some future point onto different nodes. SIMPL-Python can work seamlessly on heterogeneous networks provided at least one node is Linux. bob -- http://mail.python.org/mailman/listinfo/python-list
