Cem - I just wanted to say I agree that MessagePack and Protobuf are both very capable serdes libraries - I've worked with them both before. Some languages don't deal that well with JSON and they're excellent choices.
On Sat, Aug 18, 2012 at 12:32 PM, Cem Karan <[email protected]> wrote: > I agree with Brian that using a serialization library is a good idea, but > they'll only verify that the archived byte string can be deserialized > correctly. To verify that your system won't get into a wedged state, > you'll need to use something stronger, such as spin (thank you for > mentioning it Andrew!). > > Also, how concerned are you with malicious actors? Guarding against > random corruption is pretty easy, but if you know that there will be > malicious actors on your network, then you really DO need something that > can verify your model, AND you need to spend some time designing a real > threat model. That is, what capabilities will the malicious actors have? > Can they only corrupt packets, or can they see the state of the entire > network, and inject/destroy packets at will? Is there more than one > attacker? Do they have an unlimited amount of power/energy, or do they > have to make choices about when/where they attack? > > Finally, I'm going to mention MessagePack (http://msgpack.org/) and > Google Protocol Buffers (https://developers.google.com/protocol-buffers/) > as alternatives to JSON. They are only serialization libraries, but I've > noticed that MessagePack + blosc (http://blosc.pytables.org/trac) make > for a fairly powerful way of serializing & compressing data. I won't post > any numbers on transmission rates because the rest of my code is > sufficiently horrible that it will skew the results, but even with HIGHLY > non-optimal code, I'm pleased with the speeds/compression ratios I'm > getting. > > Thanks, > Cem Karan > > On Aug 17, 2012, at 7:57 AM, Brian Knox wrote: > > > Andrea: > > > > We heavily utilize JSON for our communications protocols. In cases > where we are using things such as MDP, which is multi-frame ( > http://rfc.zeromq.org/spec:7), we use JSON for the message body of the > multi frame message. > > > > I see you are using Python - a really nice library for designing message > "models" that can be serialized to JSON in python is "structures": > > > > https://github.com/j2labs/structures > > > > It does type validation, which is very handy for ensuring your messages > are what you expect. > > > > Additionally, the "ujson" json library for Python is pretty fast: > > > > http://pypi.python.org/pypi/ujson/ > > > > We combine this with multi frame protocols because we do not want to > have to parse the JSON frame for things like routing information. > > > > Brian > > > > On Fri, Aug 17, 2012 at 7:29 AM, andrea crotti < > [email protected]> wrote: > > 2012/8/17 Andrew Hume <[email protected]>: > > > yes! > > > you have discovered that devising a distributed computation is almost > > > isomorphic > > > to designing the protocol the distributed parts use to communicate. > > > > > > this is, in general, quite hard, and i know of no useful shortcuts > other > > > than > > > to try and build on existing patterns (such as in teh Guide). > > > i have used one method which, although somewhat tedious, did actually > work. > > > > > > the protocol part of teh code was written in pseudocode with m4 macros. > > > when you "compiled" the pseudocode, it produced two outputs: > > > 1) working C code > > > 2) working Spin code (this language used to be called Promela). > > > > > > Spin is a protocol verifier which can handle fairly large protocols. > > > the build process verified that the protocol was "correct" (taht is, > > > it had no deadlocks etc) and then built the executable. > > > > > > there were occasional missteps in ensuring the two parts were > equivalent, > > > but i will say we never had a protocol botch. that is, we never > deadlock'ed > > > or wedged because of a protocol error. these had all been caught > > > during compiling. > > > > > > obviously, none of this was specific to spin; any protocol verifier > would > > > work, i think. > > > > > > > Uhm that's really interesting, I used spin a long time ago in > > university but I didn't even think it might be useful now, and it's > > probably a very neat idea to formally verify the protocols.. > > > > Would you mind to share a simple example maybe? I will use Python and > > might generate the spin code from there too, but would still be nice > > to see something running.. > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
