Hi Nathan,
I'm going to chop this into 2 threads if that's OK, to make it more manageable. I'll chop the outbox -> inboxes & mascot stuff into here to leave the syntactic suggestions in a separate thread, so that they're easier to follow. On Monday 27 July 2009 05:46:48 Nathan Davis wrote: > Thanks for the response, Michael. First off, I think it would be helpful > to explain where I'm coming from. One of my major interests is image > processing, and I have a modest library of routines to perform various > tasks (enhancement, segmention, labelling, etc.). (It's publicly available > at https://launchpad.net/pimp if anyone's interested.) What I would like > to have is a way to "wire" these routines together. I see the link :-) > I currently have a system based on generators that can be used to chain > processors together into a pipeline. This works surprisingly well, but has > the significant shortcoming that it can not represent arbitrary graphs -- > it's pretty much restricted to pipelines. Yep. You can munge pure generators in other ways, but it gets messy when you do, IMO. (processing flow gets tied up with dataflow, which can lead to subtle bugs) > Most of the time, a DAG would probably be sufficient. Some cases may call > for feedback loops, however. Indeed. > It's also worth noting that a single output may be connected to more than > one input. I bring this up mainly because doing this in Kamaelia seems to > require the use of Backplanes. Ah. Yes. The situation where you send a single image down two different processing things before later combining them. There's lots of technotes at work based around this sort of thing with that sort of diagram :-) Incidentally, the reason for one outbox -> one inbox rule in kamaelia is based around making the system more deterministic. Backplanes are useful for this sort of thing, but aren't the only mechanism for handling this. Also I don't think current code is the final example of this sort of thing either! Current options for one outbox -> multiple inboxes are found here (in rough order that they were written) : 1 Kamaelia.Util.Fanout : Fanout 2 Kamaelia.Util.Splitter : Splitter 3 Kamaelia.Util.Splitter : PlugSplitter & Plug 4 Backplane 4 actually uses 3 underneath the hood, and 2 is really a simplified version of 3. There was an older version, which is still in Sketches that looks like this: class Splitter(Axon.Component.component): def __init__(self, outboxes): print self.__class__.Outboxes self.Outboxes = dict(self.__class__.Outboxes) self.Outboxes.update(outboxes) super(Splitter, self).__init__() def main(self): while 1: yield 1 self.pause() if self.dataReady("inbox"): data = self.recv("inbox") for box in self.Outboxes: self.send(data, box) Which is a bit hacky, but works. (in here: http://tinyurl.com/kjl6x3 ) Also more recently I was creating a load balancing based splitter here: * http://tinyurl.com/ml56tz (Uses a round robin balancing scheme - very hacky, but what was needed) Fanout for example is used like this: Graphline( source = MyDataSource(...), split = Fanout(["toConsole","toFile"]), file = SimpleFileWriter(filename="outfile"), console = ConsoleEchoer(), linkages = { ("source","outbox") : ("split","inbox"), ("split","toConsole") : ("console","inbox"), ("split","toFile") : ("file","inbox"), } ).run() Or like this: X = Graphline( newCat = newCat, rotator = loopingCounter(rotation_speed), translation = cartesianPingPong(position,screensize[0], screensize[1],border), scaler = bouncingFloat(scale_speed), imaging = continuousIdentity(cat), shutdown_fanout = Fanout(["rotator","translation","scaler", "imaging","self_shutdown"]), linkages = { ("rotator","outbox" ) : ("newCat", "rotator"), ("translation","outbox" ) : ("newCat", "translation"), ("scaler","outbox" ) : ("newCat", "scaler"), ("imaging","outbox" ) : ("newCat", "imaging"), ("newCat", "signal" ): ("shutdown_fanout", "inbox"), ("shutdown_fanout", "rotator") : ("rotator", "control"), ("shutdown_fanout", "translation") : ("translation", "control"), ("shutdown_fanout", "scaler") : ("scaler", "control"), ("shutdown_fanout", "imaging") : ("imaging", "control"), ("shutdown_fanout", "self_shutdown") : ("shutdown_fanout", "control"), } ).activate() The problem I think you'll have with this is the complexity of the graphline. Uses of the splitter or plug splitter mentioned above was really intended in the context of network servers, and so their usage is more in the case of where you need to extend or reduce the amount of fanout. More recently though, when writing the PAR component, I realised that a generic Chassis component for handling splitting would be much nicer, but with a PAR component it's very unclear what the policy should be. Is it demuxing? Is it load balancing? Is it splitting ? etc. Since I couldn't decide, I decided to leave that unsaid for that point in time: * http://tinyurl.com/nbespx And left this note in it's docs: (with a couple of corrections) PAR(inputpolicy=None, outputpolicy=None, *components) Inputs to inboxes can be controlled by passing in a policy. The default policy is this:: messages to "control" are forwarded to all children if a control message is a shutdownMicroprocess, shutdown when all children exit, exit. messages to "inbox" are currently ignored See the module docs on writing a policy function. Policies To be written. The idea behind policies is to allow someone to override the default behaviour regarding inbox data. This potentially enables the creation of things like threadpools, splitters, and general workers. The advantage this would have is then you could do this: Pipeline( FrameSource(), FrameTagger(), SplitterChassis( EdgeDetect(), MotionDetect(), ImageAnalyser(), # I dunno, I'm not an image person :-) ), Combiner(), # could take tags and use them for combining ... ) Where a splitter chassis is just a specialisation of PAR - with a policy that says "copy to all children". Having a splitter chassis like this would simplify a lot of cases IMO. If you wanted to build this now BTW, the change needed to PAR is relatively small. You'd change this: for _ in self.Inbox("inbox"): pass Here: http://code.google.com/p/kamaelia/source/browse/trunk/Code/Python/Kamaelia/Kamaelia/Chassis/PAR.py#242 To this: for msg in self.Inbox("inbox"): for c in self.components: L = self.link( (self, "_co"), (c, "inbox")) self.send( msg, "_co") self.unlink(thelinkage=L) I can see it being highly desirable to actually make these linkages once though which is a larger change - easily doable though. None of this replaces the comments you made about syntax btw, but I thought of sufficient interest to merit breaking out separately. Regarding your other points, thank you for getting back to me, I'll mull them over and reply tonight :-) FWIW, as noted I'm seriously thinking about dropping support prior to 2.5.1, with the intent of using that to improve things. I also wasn't saying that your syntax ideas weren't implementable in other languages etc, just more where I'm coming from :-) I can get quite inventive with some things, but just want to make sure I'm not making it /painful/ to switch if necessary. :-) I certainly agree for example that Graphlines can get messy, but it's kinda tricky to see how to express them better in python without resorting to code that looks like ascii art. Your suggestions as a result look like an interesting compromise :-) > By "can't read how it maps across", do you mean you are having difficulty > understanding the intent of the code, That. I'm not saying that I can't read it, but I can't read it with certainty that I'm reading it the "right" way. This is probably a factor of how I think rather than the code. I doubt you're the only person it's clear to ! :-) > I may also get in contact with Richard Taylor incidentally who has been > looking into a (very old) system called MASCOT for a while, where channels > and pools - which have remarkable similarities to inboxes & the > tracker/STM, for a summary of those. > > The reason behind that is because a) MASCOT worked b) it's been largely > forgotten, but is hugely relevant c) if we start making boxes attributes or ... > Is there a website for this? Is it open source? It's ancient (in computing terms - nearing 40 years old...) and became an open standard (or nearly did - through the BSI maybe). So old it went through a traditional/formal standardisation process *years* before the development of the web. The downside is there's very few websites on it. Search term: * Modular Approach to Software Construction Operation and Test Wikipedia page: http://en.wikipedia.org/wiki/Modular_Approach_to_Software_Construction_Operation_and_Test This page has a link to "MASCOT 2002": http://www.object-forge.com/ but has a wierd page setup preventing linkage directly to the content. It appears to have been invented by Hugo Simpson among others, and he does have a web page here: * http://async.org.uk/Hugo.Simpson/ Which gives some idea of the sheer amount of work done on the system, and you can find references in places that hold academic papers that cite MASCOT but the papers are a pain to get hold of. However, the official handbook is available here: http://async.org.uk/Hugo.Simpson/MASCOT-3.1-Manual-June-1987.pdf Which is a remarkably detailed. Comes at things from a slightly different perspective, but it's remarkable how close some parts are... Incidentally, it also looks like MASCOT was a major influence on SSADM's data flow diagrams - it could just be convergent evolution however. (SSADM always reminds me of more like s&m though) Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~----------~----~----~----~------~----~------~--~---
