Re: Need some IPC pointers
On 12/01/2011 08:03 AM, Andrew Berg wrote: I've done some research, but I'm not sure what's most appropriate for my situation. What I want to do is have a long running process that spawns processes (that aren't necessarily written in Python) and communicates with them. The children can be spawned at any time and communicate at any time. Being able to communicate with non-local processes would be nice, but is not necessary. The implementation needs to be cross-platform, but child processes will use the same OS as the parent during runtime. I don't think I'll ever need to transfer anything complicated or large - just strings or possibly tuples/lists. I'd rather not go outside the standard library (but I'd consider it). I don't need to worry about compatibility with older Python versions; if it only works with Python 3.2, that's not a problem. I'm thinking sockets, but perhaps there's something simpler/easier. Considering your requirements, I'd suggest a RESTful web service. It is fairly trivial to write HTTP clients in most languages, and python's standard library comes with a simple HTTP server so writing the server is easy as well. In context, the long running process will be the server, the children processes will be the client. Writing HTTP client is fairly trivial in most languages, the protocol is platform independent, and it is fairly trivial to communicate with non-local processes over the LAN or the Internet. As a plus, it's well standardized. As the data interchange format, I suggest either xml or json. There is a library for xml and json in almost any popular languages. Python comes with both. -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
I'm surprised no one has mentioned zeromq as transport yet. It provides scaling from in proc (between threads) to inter-process and remote machines in a fairly transparent way. It's obviously not the python stdlib and as any system there are downsides too. Regards, Floris -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
What about named pipes? I don't mind a bit of if Windows do this, else, do that as long I'm not coding two or more completely different approaches. I'm not too familiar with named pipes, though; perhaps someone with some experience could chime in. Apparently this didn't go through to Google Groups the first time; it may show up twice for some. -- CPython 3.2.2 | Windows NT 6.1.7601.17640 | Thunderbird 7.0 -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
On Nov 30, 4:03 pm, Andrew Berg bahamutzero8...@gmail.com wrote: I've done some research, but I'm not sure what's most appropriate for my situation. What I want to do is have a long running process that spawns processes (that aren't necessarily written in Python) and communicates with them. The children can be spawned at any time and communicate at any time. Being able to communicate with non-local processes would be nice, but is not necessary. The implementation needs to be cross-platform, but child processes will use the same OS as the parent during runtime. I don't think I'll ever need to transfer anything complicated or large - just strings or possibly tuples/lists. I'd rather not go outside the standard library (but I'd consider it). I don't need to worry about compatibility with older Python versions; if it only works with Python 3.2, that's not a problem. I'm thinking sockets, but perhaps there's something simpler/easier. -- CPython 3.2.2 | Windows NT 6.1.7601.17640 | Thunderbird 7.0 You might want to take a look at the SIMPL toolkit. http://www.icanprogram.com/06py/lesson1/lesson1.html SIMPL modules can be written (and mixed) in any number of supported languages including Python, C, C++, JAVA, Tcl/Tk or PHP. Non local communication is handled by generic surrogates. Because of this surrogate architecture SIMPL modules can often be tested locally and then deployed into the cloud without any changes. bob -- http://mail.python.org/mailman/listinfo/python-list
Need some IPC pointers
I've done some research, but I'm not sure what's most appropriate for my situation. What I want to do is have a long running process that spawns processes (that aren't necessarily written in Python) and communicates with them. The children can be spawned at any time and communicate at any time. Being able to communicate with non-local processes would be nice, but is not necessary. The implementation needs to be cross-platform, but child processes will use the same OS as the parent during runtime. I don't think I'll ever need to transfer anything complicated or large - just strings or possibly tuples/lists. I'd rather not go outside the standard library (but I'd consider it). I don't need to worry about compatibility with older Python versions; if it only works with Python 3.2, that's not a problem. I'm thinking sockets, but perhaps there's something simpler/easier. -- CPython 3.2.2 | Windows NT 6.1.7601.17640 | Thunderbird 7.0 -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
There are two different problems. One is the medium to pass messages, sockets are good but there are other options and depends on your requirement you need to pick the best one. The other is serialization format, here you have marshal, pickle, JSON, XML and more. For me JSON over sockets works well in the past. It has the nice property that you can read messages without need for a special decoder (like in the case of marshal/pickle). -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
I'm thinking sockets, but perhaps there's something simpler/easier. Sockets are the only thing that will work without threads on all platforms. You could also use threads and pipes. (I'm not actually sure how threads+pipes works, but I'm told that it's a viable approach). Usually things are simpler and easier because they wrap all the cross-platform messiness behind a nice API, and then give you a higher level protocol for IPC on top of that. Unfortunately, the fewer external dependencies you have, the more you have to rewrite yourself. For what it's worth, I wrote something potentially similar using Twisted and AMP. AMP is an Asynchronous Messaging Protocol: this basically means that clients and servers can send messages to each other at any time in any order. Twisted makes sure that the right response gets associated with the right message. This can be very convenient -- you might request something from another process, and then to compute its answer it might ask for some additional information from you, and then you give it that information, and it sends back the final result. All the communication is done over TCP, usually using Twisted. So this does involve bringing in a fairly large dependency. For me, the tradeoff in programmer productivity was worth it, but of course it really depends on your situation. A more synchronous protocol might be more useful to you (although my impression was that perhaps you might be interested in something like AMP), and synchronous protocols directly on top of sockets are not so difficult to implement (although making them work across platforms is still too hard for my taste). Devin On Wed, Nov 30, 2011 at 4:03 PM, Andrew Berg bahamutzero8...@gmail.com wrote: I've done some research, but I'm not sure what's most appropriate for my situation. What I want to do is have a long running process that spawns processes (that aren't necessarily written in Python) and communicates with them. The children can be spawned at any time and communicate at any time. Being able to communicate with non-local processes would be nice, but is not necessary. The implementation needs to be cross-platform, but child processes will use the same OS as the parent during runtime. I don't think I'll ever need to transfer anything complicated or large - just strings or possibly tuples/lists. I'd rather not go outside the standard library (but I'd consider it). I don't need to worry about compatibility with older Python versions; if it only works with Python 3.2, that's not a problem. I'm thinking sockets, but perhaps there's something simpler/easier. -- CPython 3.2.2 | Windows NT 6.1.7601.17640 | Thunderbird 7.0 -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
On 11/30/2011 3:32 PM, Devin Jeanpierre wrote: You could also use threads and pipes. (I'm not actually sure how threads+pipes works, but I'm told that it's a viable approach). Sounds interesting, but I'm not familiar with threading (not that I wouldn't be willing to learn). Is it even possible to pipe into a running process, though? For what it's worth, I wrote something potentially similar using Twisted and AMP. AMP is an Asynchronous Messaging Protocol: this basically means that clients and servers can send messages to each other at any time in any order. Twisted makes sure that the right response gets associated with the right message. This can be very convenient -- you might request something from another process, and then to compute its answer it might ask for some additional information from you, and then you give it that information, and it sends back the final result. All the communication is done over TCP, usually using Twisted. So this does involve bringing in a fairly large dependency. Sounds like overkill, but I'll take a look. -- CPython 3.2.2 | Windows NT 6.1.7601.17640 | Thunderbird 7.0 -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
On 30-11-11 22:03, Andrew Berg wrote: I've done some research, but I'm not sure what's most appropriate for my situation. What I want to do is have a long running process that spawns processes (that aren't necessarily written in Python) and communicates with them. The children can be spawned at any time and communicate at any time. Being able to communicate with non-local processes would be nice, but is not necessary. The implementation needs to be cross-platform, but child processes will use the same OS as the parent during runtime. I don't think I'll ever need to transfer anything complicated or large - just strings or possibly tuples/lists. I'd rather not go outside the standard library (but I'd consider it). I don't need to worry about compatibility with older Python versions; if it only works with Python 3.2, that's not a problem. I'm thinking sockets, but perhaps there's something simpler/easier. Standard library, local processes: multiprocessing module. If that doesn't suit your needs, maybe check out Pyro: http://packages.python.org/Pyro4/ Pyro allows objects to talk to each other over the network, with minimal programming effort. You can just use normal Python method calls to call objects on other machines (or locally, ofcourse). It's written in 100% pure Python and works on Python 2.6 and upwards (including 3.x). Regards, Irmen de Jong -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
Sounds interesting, but I'm not familiar with threading (not that I wouldn't be willing to learn). Is it even possible to pipe into a running process, though? You create the pipe to the process when you start it. e.g. subprocess.Popen(['ls', 'foo'], stdout=subprocess.PIPE, stdin=subprocess.PIPE) Irmen de Jong mentions the multiprocessing module, that might be a better approach. I don't use it for production purposes because it has bad properties when processes are killed abruptly (it can't tell necessarily the difference between a truncated message and a normal message). It doesn't feel safe, you know? (Of course, neither do threads!) Devin On Wed, Nov 30, 2011 at 5:19 PM, Andrew Berg bahamutzero8...@gmail.com wrote: On 11/30/2011 3:32 PM, Devin Jeanpierre wrote: You could also use threads and pipes. (I'm not actually sure how threads+pipes works, but I'm told that it's a viable approach). Sounds interesting, but I'm not familiar with threading (not that I wouldn't be willing to learn). Is it even possible to pipe into a running process, though? For what it's worth, I wrote something potentially similar using Twisted and AMP. AMP is an Asynchronous Messaging Protocol: this basically means that clients and servers can send messages to each other at any time in any order. Twisted makes sure that the right response gets associated with the right message. This can be very convenient -- you might request something from another process, and then to compute its answer it might ask for some additional information from you, and then you give it that information, and it sends back the final result. All the communication is done over TCP, usually using Twisted. So this does involve bringing in a fairly large dependency. Sounds like overkill, but I'll take a look. -- CPython 3.2.2 | Windows NT 6.1.7601.17640 | Thunderbird 7.0 -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
On 30-11-11 23:28, Irmen de Jong wrote: On 30-11-11 22:03, Andrew Berg wrote: processes (that aren't necessarily written in Python) and communicates Oops, missed this on my first read. This rules out my suggestion of Pyro because that requires Python on both ends (or Java/.net on the client side). Sorry for the noise. Irmen -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
On Thu, Dec 1, 2011 at 8:03 AM, Andrew Berg bahamutzero8...@gmail.com wrote: processes (that aren't necessarily written in Python) ... non-local processes would be nice ... The implementation needs to be cross-platform ... I don't think I'll ever need to transfer anything complicated or large - just strings or possibly tuples/lists. I'm thinking sockets, but perhaps there's something simpler/easier. Definitely sockets, as other posters have said. The only question is, what encapsulation format (since sockets give just a stream of bytes). Since you want non-Python processes to be involved, I would be inclined to avoid using Python's own pickling system; JSON may be viable, and it's well known so you should be able to find support in other languages. Alternatively, take a careful look at your dataset and invent your own system. If your strings will never contain a pipe character, you could define a tuple/list to be simply pipe-separated strings; if they'll never contain newlines, you can specify your protocol to be newline-terminated. For command/response protocols over TCP/IP, I strongly recommend poking around with existing protocols such as the email trio (SMTP, POP, and IMAP); there's many good ideas to be gained from them. If you can, try to keep your protocol to ASCII text. It makes debugging far easier, as you can simply point a telnet/mud client at your server and manually step through things. Chris Angelico -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
Sure, I'll give you some pointers: 0x3A28213A 0x6339392C 0x7363682E -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
In article mailman.3188.1322714125.27778.python-l...@python.org, Alec Taylor alec.tayl...@gmail.com wrote: Sure, I'll give you some pointers: 0x3A28213A 0x6339392C 0x7363682E What, no 0xDEADBEEF ??? -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
:P On Thu, Dec 1, 2011 at 4:02 PM, Roy Smith r...@panix.com wrote: In article mailman.3188.1322714125.27778.python-l...@python.org, Alec Taylor alec.tayl...@gmail.com wrote: Sure, I'll give you some pointers: 0x3A28213A 0x6339392C 0x7363682E What, no 0xDEADBEEF ??? -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Need some IPC pointers
On 11/30/2011 10:35 PM, Alec Taylor wrote: Sure, I'll give you some pointers: I almost rephrased the subject line because I knew someone would make that joke. :P -- CPython 3.2.2 | Windows NT 6.1.7601.17640 | Thunderbird 7.0 -- http://mail.python.org/mailman/listinfo/python-list
