Distributed pipes are an interesting idea. I started work on something similar, but from the other direction, wrapping processes and capturing or teeing their stdin/out and other metadata (memory, cpu usage) and sending that up to a central broker. This 'nerve' process wrapper mimics the Python Popen interface and is as transparent as I could make it.
https://github.com/michelp/nerve Nerves talk to centers (N dealers to a router) and centers can be accessed with a command line interface to query the state of the various processes being managed by nerves. In a sense it's very similar to the popular 'supervisor' package that is commonly used to watchdog processes, but can manage nerves distributed across machines. I wanted to extend the notion so that a cluster of centers could exchange state with each other (what procs/streams they are managing) and connect their various nerve wrapped processes input and outputs together to make distributed pipes. Distributed process management was my main goal, but distributed pipes is a natural unixy place to go with that. I didn't get much farther with the idea. The prototype is in Python and uses pyzmq and gevent. -Michel On Wed, Apr 16, 2014 at 11:35 AM, Pieter Hintjens <[email protected]> wrote: > Hi all, > > I'm working on a distributed pipes service (in zeromq/zbroker), and > wonder if anyone would like to try this in their work. > > The architecture is: > > * Client applications read/write from named pipes > * These pipes are hosted on edge brokers, on same box as clients. > * Clients talk to brokers over IPC (ironic, but so it is). > * Brokers interconnect and chat over Zyre. > > I'm working out the semantics for N-to-N pipes and may go for a pubsub > model, see https://github.com/zeromq/zbroker/issues/38. > > -Pieter > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
