Thanks for this, Michael. It would be great to see examples of cross-machine IPC, even though I realize it amounts to the same classic fork and poll/select socket I/O model of IPC on a single machine/processor. People generally want to see IPC processes started at different times, with some form of built-in load balancing. Maybe a few of us should give you ssh and some open ports on our machines? I have a publicly addressable machine, and can do this. Let me know if you want access to it. Or maybe we should ping some universities for access to a chunk of their resources for this purpose. Gloria > On Monday 08 June 2009 18:38:40 Gloria W wrote: > >> I am still, however, stuck on the last series of messages regardin >> interprocess communication. It wasn't clear to me if you would support the >> Python 2.6 processing module, >> > > At some point, yes. (Hopefully sooner rather than later, but I don't want to > pin a date on it.) > > >> or if your interprocess communication is DIY. >> > > For interprocess communication inside a system (ie single source file) you > use > one of these from Axon.experimental.Process : > * ProcessGraphline > * ProcessPipeline > * ProcessPipelineComponent > * ProcessWrapComponent > > Under the hood these are implemented using Paul Boddie's pprocess library > from > here: > * http://pypi.python.org/pypi/pprocess > > These all allow you to use multiple processes (and hence processors), because > in the words of Paul's description, pprocess: > "The pprocess module provides elementary support for parallel > programming in Python using a fork-based process creation model in > conjunction with a channel-based communications model implemented > using socketpair and poll." > > Under the hood, this boils down to: > * os.fork for splitting into multiple processes > * os.pipe's for communication between them > * cPickle for passing objects between processes. > > This naturally allows a container component to fork to distribute > subcomponents to other processes. > > >> Can you please point me to some more recent examples of IPC in Kamaelia? >> > > It's important to realise something here. This allows you to put *ANY* > components in these containers. As a result, almost _any_ Kamaelia system is > a candidate for demonstrating this. For example taking this from yesterday: > Pipeline( > Textbox(size = (800, 300), position = (100,380)), > InterpreterTransformer(), > TextDisplayer(size = (800, 300), position = (100,40)), > ).run() > > Turning this into a system that uses 4 processes (one container + 3 sub > processes), is as simple as this: > > from Axon.experimental.Process import ProcessPipelineComponent > > ProcessPipelineComponent( > Textbox(size = (800, 300), position = (100,380)), > InterpreterTransformer(), > TextDisplayer(size = (800, 300), position = (100,40)), > ).run() > > I've checked this in here: > http://code.google.com/p/kamaelia/source/browse/trunk/Code/Python/Kamaelia/Examples/PythonInterpreter/MultiProcessPygamePythonInterpreter.py > > The proof of this is naturally the fact that pygame is limited to one SDL > window per process at present and of course this opens 2 such windows. > > It's worth mentioning though that the process behaviour of this example is > exactly equivalent to a normal unix pipeline. (each part in it's own sub > process, communicating serialised data over an os.pipe). > > ... and just like a normal unix pipeline, the components *in* the pipeline > neither know, or nor need to care about this. > > >> This is the most common question asked of me when I speak about Kamaelia >> publicly. >> > > I do agree we need more examples in this area, but once you have the model > right, this does mean that every example is an example of IPC... > > I doubt this is a very satisfactory answer though (unless it causes a > lightbulb moment... :) so I will say this: I'm likely to get this question > myself in just under 3 weeks time at Europython since I'll be giving a > Kamaelia tutorial there (in Birmingham UK), so I'll try to provide a nicer > example for it there, and hence for here on the list too. > > There isn't however an example at present for this example: > * Multiple machines come together in a server farm > * Register themselves as processing slaves > * Then a system is able to distribute their work over this system. > > ... with the primary reason being *I don't have a server farm* :-) > > This would probably make a nice sprint to do at some point, since we have > most of the components you'd need to make this happen. (eg multicast your > presence on the network, allows nodes to autodiscover each other, then > perform some validation of nodes, and then use a model a bit like > ProcessPipeline, but linked in with the PythonInterpreter code build a job > server/handler.) Drawing a diagram would probably a good starting point for > that :) > > Thanks for reminding me that it does need addressing, reminders of the > questions that are most often asked are very useful! :) > > Regards, > > > Michael. >
--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~----------~----~----~----~------~----~------~--~---
