When we get to looking at Python I'll have a chance to play around a bit more. Our wire-format is pretty basic, but based around serialized protobufs, so it should be simple to swap out codecs, as long as there is a clean way to access that part of the framework.
Cheers! Matt On Nov 29, 2011, at 7:58 PM, MinRK wrote: > > > On Tue, Nov 29, 2011 at 00:50, Matthew Long <[email protected]> wrote: > As a short followup -- I just looked at PyZMQ and it looks like an excellent > "pythonic" binding. In particular, it handles asynchronous calls and > queueing calls so that they can be sent via a single socket thread. > > Glad you like it! The tornado IOLoop/ZMQStream stuff in zmq.eventloop should > make writing a Python RPC server pretty straightforward. The model we use in > IPython.parallel is essentially Async RPC-ish, and we use JSON to serialize > by default, but allow drop-in replacement of other serialization methods for > performance/language compatibility. I've had good experience with msgpack, > and I believe protobuf works as well. > > -MinRK > > > Unfortunately, I am not doing anything in Python at the moment. > > Cheers, > > Matt > > On Nov 29, 2011, at 9:29 AM, Matthew Long wrote: > > > Hi Pieter, > > > > On Nov 28, 2011, at 10:03 PM, Pieter Hintjens wrote: > > > >> On Mon, Nov 28, 2011 at 8:36 AM, Jakub Witkowski <[email protected]> wrote: > >> > >>> On the other hand, making a "RPC-like" system, fully customized to your > >>> application based on ZMQ is very simple; it more or less involves > >>> copy-pasting the dealer/router pattern from the examples and putting in > >>> your business logic in. Toss in Google Protocol Buffers and suddenly you > >>> have a full featured solution that is both easy to write and very, very > >>> fast. > >> > >> Yes, this is what I'd recommend as well. Google protobufs give you > >> language portability, are fast, and easy to use. There are various RPC > >> examples in the Guide you can start with, see Ch3 and Ch4. I'd suggest > >> either the Majordomo or Freelance patterns. > > > > Yes, those are a good start. We already use protobufs for the IDL, service > > definitions and serialization. > > > > Where these examples start breaking down when I try to bring them into my > > existing frameworks: > > 1) Sockets are not thread safe so we need to build scaffolding to protect > > the socket. When threads are expensive like in C/C++/etc it makes sense to > > do this via inproc:// connections, but in other languages (such as Go) > > where go routines can be multiplexed onto any number of running threads (by > > the runtime) this is an issue. > > 2) Clients often want asynchronous messages and callbacks. That is, the > > client might send requests 1, 2, 3, 4, 5, but get back 5, 3, 4, 1, 2 -- and > > we need to pair the request context with the correct response. > > 3) Go is a typesafe language, but we want to be able to do the type > > conversion when we serialize and deserialize messages. > > > > All three are already handled in the RPC layer -- the fact that I was > > reimplementing most of this is what prompted my post to the list. > > > > What I am hearing is there are no existing tools to do this and I should > > implement something myself. > > > > It may be that with the appropriate surgery I can fit 0MQ in with existing > > frameworks -- but this is something that the 0MQ team might want to > > consider addressing at some point. While there are language bindings in a > > ton of languages, a strict port of the C api probably won't be idiomatic in > > the target language. If 0MQ fit in easily to the right layer of Go or Java > > or Python it would be a clear win. I could use 0MQ with the native RPC > > system and also implement neat patterns on top of it for PUB/SUB or > > whatever. As it stands it is a little bit of a mixed win for us. > > > > If I do figure out how to cleanly add 0MQ to various RPC systems, I will > > post what I did to the list. > > > > Matt > > > >> > >> -Pieter > >> _______________________________________________ > >> zeromq-dev mailing list > >> [email protected] > >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
