I am developing a new application using zmq in an unusual (from what I've seen so far) way. I'd like to solicit input, particularly about reasons this may be a terrible idea. It is essentially a variation on an event-driven architecture.
The application essentially collects data from a variety of sources, does various sorts of processing on the data, and reacts to data (including both raw data and at various stages of processing) in various ways, including taking actions, reporting on action results (i.e., injecting new data), or emitting events. This application, unlike many that fit the above description, is NOT concerned with distributing the workload among multiple nodes for scaling or reliability. The fact that I'm using zmq means I could distribute it, but in this case there's really no reason (so far). The volume of data and events is quite small. I am not looking for scaling as much as I'm looking for making it easy to develop and build an application with is flexible and robust in the face of complex and changing requirements. My tentative solution is to locally run a "broker" which does nothing other than re-broadcast all messages it receives on a SUB socket to a PUB socket. It binds both the SUB and the PUB (to different ports), as opposed to the typical pattern of binding PUB and connecting the SUB to it. I then run a bunch of small, single-purpose daemons (call them "agents" if you prefer) which connect their own SUB to the broker's PUB to hear messages, and their own PUB to the broker's SUB in order to broadcast messages. Basically, this creates an environment where all these daemons are shouting into the same room, and everybody hears everybody else, including getting their own messages fed back to themselves. Each daemon can filter this by putting a filter on their SUB socket if they like. Unlike in some event-driven architectures, there is no concept of "handling" an event or request in a way that halts further processing by other "handlers". The event gets shouted into the room, and anybody that cares to do something with it can. Like a noisy gymnasium, you mostly have small groups of people playing at different games with different balls, but you can also sit in the bleachers and watch everything all at once. The advantages I see are: Decoupling: Unit testing is trivial, and you are encouraged by the architecture to divide the work into very small, very simple, independently reusable routines that each do one and only one thing. Transparency: It is easy to trace and log a system like this. Unlike a system where handlers can "consume" events that then nobody else gets to act on, in this system multiple daemons can act on every message. This makes it easy to implement rules like "No matter what else happens, if X happens, do this." In fact, every rule/processing step you implement in a system like this will be of that form; every matching rule always fires. This, I believe, makes the system more flexible, because today you may think you know how to "handle" an event, but tomorrow you will find that you need to do all of that, plus another 10 things that you didn't know about yesterday. If you have a system where events are explicitly handled, you need to go back to that single event handler and add those 10 things to it. Configurability: Continuing that thought, if you need to run different configurations of the application which have different sets of rules, it is easy to do without ending up with huge event handlers that have a bunch of conditional statements in them based on settings from a configuration file or database. Instead, in each environment, you either run the daemon that implements a given rule/process, or you do not run that daemon. The source code for the daemons themselves can then be much simpler and uncluttered, with little if any configuration involved. Language independence: Code each rule/process in the language that makes the most sense for that individual piece. This is actually the most trivial benefit for me, but possibly quite useful for others. What are the downsides? Scalability, obviously, because rebroadcasting all events to everybody is not going to be very performant with either very high volume of messages or very large numbers of cooperating daemons. Reliability may also be an issue; obviously PUB/SUB can drop messages, and if a given daemon is not up and running when a message comes across the wire, then whatever that daemon was supposed to do with it is not going to happen. But the idea is that this is all happening on localhost, or at least a reliable LAN, and that there will be some supervisor process that first starts the broker and then everything else, and restarts dead daemons as necessary. Still, this is not a solution with 100% reliability. Anybody else have other thoughts? Is this is bad idea? A useless idea? Better accomplished another way? Any "Yeah, I did something like that once -- never again" stories? _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
