Hey, Don't worry, this isn't really about how I do things personally. I totally agree that there is some extra overhead in maintaining the Protobuf/Thrift approach over just passing around json. Obviously one should pick the best format for the job and json could easily be the right choice for an application. But the same things that could be looked at as negatives could also be looked at as positives. Typing out the proto file acts as your 'template', and could be used to auto-document your structure. And it could actually save you typing if you are dealing with structures that have default values.
Consider the example from the protobuf tutorial: https://developers.google.com/protocol-buffers/docs/pythontutorial Now you could distribute your packages with the python code already generated and not distribute the proto file. That can be just part of your source distribution. Building the proto file again only happens if you change things in your spec. So once you have that example built, it could look like this: from addressbook_pb2 import * me = Person(name='Justin', id=1, phone=[Person.PhoneNumber(number='555-1212')]) And because the spec already knows about default values, it has already set my phone number type to HOME, and my email field is an empty string. The equivalent pure json approach would require something like this: import json HOME=1 me = { "name": "Justin", "id": 1, "email": "", "phone": [ {"number": "555-1212", "type": HOME } ] } me_msg = json.dumps(me) So I would say in this case, at least to my eyes, it is slightly less work and more descriptive to define a Person protobuf vs the python dict And on top of that, if I were to try and pass a number as my name, or a string for my id, the Person object would immediately raise an exception. If I were using this json approach, I would probably end up writing a function or class that abstracts the format anyways, so anyone using my library would have something easier and more consistent. Writing the proto file then pretty much has the same effort as writing abstraction classes or functions to produce your message format in json. Like I said, both formats can work for different needs and they both have benefits. On Wed, May 21, 2014 at 5:42 PM, Marcus Ottosson <[email protected]>wrote: > Hey Justin, > > I hope this isn’t coming across as criticism for how you do things > personally (as I take it you are working with something similar using a > similar approach) but more as exploring the various approached in which > commands can be sent across processes with the intent on weighing the > benefits with the risks. > > Why do you care if a command message is human readable? The wire protocol > should not matter to the user creating a message. They would go through an > interface either way that produces the message transparently to them. > > If I understand you correctly, you are referring the the end-user of > structured messages; I’m referring to the developer of them. Someone will > have to read and write the code that make up the protocol, and for that > person, there is obviously more work involved *up-front*. > > E.g. it takes more research and more typing to write a protobuf, a message > using protobuf and a build script to automatically build the protobuf, then > it does in simply writing a in-line dict and send it across. > > Again, the goal is not to prove you or anyone else right or wrong. I’m > looking for concrete examples of where one is more suitable over the other. > > > > On 20 May 2014 22:06, Justin Israel <[email protected]> wrote: > >> Ok then, so we just focus on the third point. >> >> Why do you care if a command message is human readable? The wire protocol >> should not matter to the user creating a message. They would go through an >> interface either way that produces the message transparently to them. >> >> >> On Wed, May 21, 2014 at 8:30 AM, Marcus Ottosson >> <[email protected]>wrote: >> >>> Ok, now we're talkin'. :) >>> >>> The "faster" statement is indeed ambiguous and could mean one in three >>> things; >>> >>> 1. It is fast to ship over the wire >>> 2. It is fast to marshall >>> 3. It is fast to work with, as a human >>> >>> I am referring to number 3, because in command-land, messages are >>> neither marshalled or transferred often enough to make any significant >>> difference in terms of performance. Let's say around command is being >>> requested once every second, on average, by multiple clients on a daily >>> basis, and that, for the sake of familiarity, the commands are similar to >>> "render me this" or "render me that" >>> >>> For 1 and 2 to be relevant, messages would probably have to be sent by >>> the thousands or even millions; which is of course often the case, but >>> possible not in the context of the Command Pattern. >>> >>> So, in terms of human performance, I stand by my statement that working >>> with a structured message is slower, yet less prone to (again, human) error. >>> >>> Thoughts? >>> >>> On Tuesday, May 20, 2014, Justin Israel <[email protected]> wrote: >>> >>>> I think you have made a completely inaccurate statement here. Or at >>>> least one that is ambiguous. You referred to a structured message as >>>> slower, and an unstructured message as faster because "it requires no >>>> parsing". >>>> >>>> When you send the json message across the wire, it will be a single >>>> message that will be read entirely and then it has to go through the json >>>> load process to be parsed and turned into an object on the other end. It >>>> also can only represent a few types in its standard spec. >>>> >>>> Protobuf has a binary format which can be smaller in size and faster to >>>> read because as it reads each index it can lookup the type from the proto >>>> and know what to read for the value. >>>> >>>> Have you done tests that suggest this fast/slow categorization is >>>> valid? Depending on your message size it could either be true, false, or >>>> negligible >>>> https://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking >>>> >>>> In that test, protobuf ended up being smaller and faster overall and >>>> was only slowest in the object creation component of the test. >>>> >>>> I'm just mentioning this because it immediately struck me as strange >>>> that you would represent the options by this criteria (fast/error prone, >>>> slow/less error prone). It usually depends on your message structure and >>>> size and required profiling. >>>> On May 21, 2014 3:59 AM, "Marcus Ottosson" <[email protected]> >>>> wrote: >>>> >>>>> >>>>> - >>>>> >>>>> Unstructured is fast, more prone to error >>>>> - >>>>> >>>>> Structured is slow, less prone to error >>>>> >>>>> I should say, “..prone to *human* error”. The computer doesn’t care >>>>> either way. >>>>> >>>>> >>>>> >>>>> On 20 May 2014 16:57, Marcus Ottosson <[email protected]> wrote: >>>>> >>>>> Yes you are probably right. >>>>> >>>>> This topic is about JSON versus Protobuf for transmitting commands >>>>> across processes; let’s refer to these as *unstructured* versus >>>>> *structured* respectively. To be explicit, and for those not familiar >>>>> with protobuf or schemas and what not, here’s what we’re comparing: >>>>> >>>>> - >>>>> >>>>> Unstructured is fast, more prone to error >>>>> - >>>>> >>>>> Structured is slow, less prone to error >>>>> >>>>> Example >>>>> >>>>> *unstructured - client* >>>>> >>>>> >>> message = { >>>>> 'command': 'create', >>>>> 'args': ['age', '5'], >>>>> 'id': 'uuid' # Unique id for client >>>>> }>>> packed_message = json.dumps(message)>>> >>>>> socket.send(packed_message)>>> return_value = socket.recv()# Blocks until >>>>> server responds>>> # No post-processing required >>>>> >>>>> *unstructured - server* >>>>> >>>>> >>> packed_message = socket.recv()>>> message = >>>>> >>> json.loads(packed_message)>>> command = message['command'] >>>>> >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBr0gSNLeDMk2uVZfK-s_c_g5w%3D88%3DU9u3BzmoA%3DNaLmw%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBr0gSNLeDMk2uVZfK-s_c_g5w%3D88%3DU9u3BzmoA%3DNaLmw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Python Programming for Autodesk Maya" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0pHAAgzw2zjN%2Bj0MQgsdMYh-cRL4rRQttWYLAu21OTWw%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0pHAAgzw2zjN%2Bj0MQgsdMYh-cRL4rRQttWYLAu21OTWw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> -- >>> *Marcus Ottosson* >>> [email protected] >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Python Programming for Autodesk Maya" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAMGtKXS6%3DJvKxJChsTSkBce1O6VkB5E_3eiYrtCm590Q%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAMGtKXS6%3DJvKxJChsTSkBce1O6VkB5E_3eiYrtCm590Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Python Programming for Autodesk Maya" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0FW-KWDM8wS2Cs70gp7mo7qxnCxk4qAzdG8hLHdd66Fw%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0FW-KWDM8wS2Cs70gp7mo7qxnCxk4qAzdG8hLHdd66Fw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > *Marcus Ottosson* > [email protected] > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBUotm6YH8OLa%2B7yqrmjO6s4JB2mpQ%3DduY2K0ZccuOZAA%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBUotm6YH8OLa%2B7yqrmjO6s4JB2mpQ%3DduY2K0ZccuOZAA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0%3Da6pYj6wbm47YKYaoaPjtnbyVCYuuo_O2LiLWrjxZYw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
