Thanks, Justin. That is some good info.
I can see two branches of our discussion; one in which clients and servers
are tightly connected, say in a corporate environment where everyone shares
libraries and network, such as protobuf. The other is communicating with an
unknown server (e.g. across the internet), one which provides a service of
which we know little about.
I think our current topic lies in the former arena; the corporate one. And
I think this is where I’m mostly interested in investigating too, so I’ll
skip the cons of working with an unknown server in regards to your example.
The only con I can think of at this point is the additional dependency;
both technically and in terms of learning. I suspect that as I get deeper
into messaging, problems will arise, problems that libraries such as
protobuf is designed to solve. But at this point, the extra work involved
is simply overwhelming compared to the few benefits gained.
For clarity, I’ll list the ones I gathered here.
Protobuf:
- Adds a learning curve
-
Adds a dependency
-
Requires an extra build step
And what you get is:
- Default values
- Type-checking, with exceptions
- Protocol versioning, i.e. safer to change
Each of which fades away when compared to figuring out bigger beasts such
as figuring out the overall architecture of an application and its
communication channels across a network or whether or not to use REQ/REP,
PUB/SUB, PUSH/PULL etc. for network communication. Before handling such
issues, it seems unreasonable also drag around extra build steps, learning
curves and dependencies, when the benefits of doing so aren’t blossoming
until later in the game.
I think that is what I’m getting at with a mature versus immature API. Not
just the syntax of the commands, but which commands it should include, and
whether or not the API is solving a real issue to begin with. I think
libraries such as protobuf can only really show its true colors once you’ve
already gained traction and can start thinking about optimisations and
maintenance.
Thoughts?
Best,
Marcus
On 21 May 2014 09:56, Justin Israel <[email protected]> wrote:
> Hey,
>
> Don't worry, this isn't really about how I do things personally. I totally
> agree that there is some extra overhead in maintaining the Protobuf/Thrift
> approach over just passing around json. Obviously one should pick the best
> format for the job and json could easily be the right choice for an
> application. But the same things that could be looked at as negatives could
> also be looked at as positives. Typing out the proto file acts as your
> 'template', and could be used to auto-document your structure. And it could
> actually save you typing if you are dealing with structures that have
> default values.
>
> Consider the example from the protobuf tutorial:
> https://developers.google.com/protocol-buffers/docs/pythontutorial
>
> Now you could distribute your packages with the python code already
> generated and not distribute the proto file. That can be just part of your
> source distribution. Building the proto file again only happens if you
> change things in your spec.
>
> So once you have that example built, it could look like this:
>
> from addressbook_pb2 import *
>
> me = Person(name='Justin', id=1,
> phone=[Person.PhoneNumber(number='555-1212')])
>
>
> And because the spec already knows about default values, it has already
> set my phone number type to HOME, and my email field is an empty string.
> The equivalent pure json approach would require something like this:
>
> import json
>
> HOME=1
>
> me = {
> "name": "Justin",
> "id": 1,
> "email": "",
> "phone": [ {"number": "555-1212", "type": HOME } ]
> }
>
> me_msg = json.dumps(me)
>
>
> So I would say in this case, at least to my eyes, it is slightly less work
> and more descriptive to define a Person protobuf vs the python dict
> And on top of that, if I were to try and pass a number as my name, or a
> string for my id, the Person object would immediately raise an exception.
>
> If I were using this json approach, I would probably end up writing a
> function or class that abstracts the format anyways, so anyone using my
> library would have something easier and more consistent. Writing the proto
> file then pretty much has the same effort as writing abstraction classes or
> functions to produce your message format in json.
>
> Like I said, both formats can work for different needs and they both have
> benefits.
>
>
>
>
> On Wed, May 21, 2014 at 5:42 PM, Marcus Ottosson
> <[email protected]>wrote:
>
>> Hey Justin,
>>
>> I hope this isn’t coming across as criticism for how you do things
>> personally (as I take it you are working with something similar using a
>> similar approach) but more as exploring the various approached in which
>> commands can be sent across processes with the intent on weighing the
>> benefits with the risks.
>>
>> Why do you care if a command message is human readable? The wire protocol
>> should not matter to the user creating a message. They would go through an
>> interface either way that produces the message transparently to them.
>>
>> If I understand you correctly, you are referring the the end-user of
>> structured messages; I’m referring to the developer of them. Someone will
>> have to read and write the code that make up the protocol, and for that
>> person, there is obviously more work involved *up-front*.
>>
>> E.g. it takes more research and more typing to write a protobuf, a
>> message using protobuf and a build script to automatically build the
>> protobuf, then it does in simply writing a in-line dict and send it across.
>>
>> Again, the goal is not to prove you or anyone else right or wrong. I’m
>> looking for concrete examples of where one is more suitable over the other.
>>
>>
>>
>> On 20 May 2014 22:06, Justin Israel <[email protected]> wrote:
>>
>>> Ok then, so we just focus on the third point.
>>>
>>> Why do you care if a command message is human readable? The wire
>>> protocol should not matter to the user creating a message. They would go
>>> through an interface either way that produces the message transparently to
>>> them.
>>>
>>>
>>> On Wed, May 21, 2014 at 8:30 AM, Marcus Ottosson <[email protected]
>>> > wrote:
>>>
>>>> Ok, now we're talkin'. :)
>>>>
>>>> The "faster" statement is indeed ambiguous and could mean one in three
>>>> things;
>>>>
>>>> 1. It is fast to ship over the wire
>>>> 2. It is fast to marshall
>>>> 3. It is fast to work with, as a human
>>>>
>>>> I am referring to number 3, because in command-land, messages are
>>>> neither marshalled or transferred often enough to make any significant
>>>> difference in terms of performance. Let's say around command is being
>>>> requested once every second, on average, by multiple clients on a daily
>>>> basis, and that, for the sake of familiarity, the commands are similar to
>>>> "render me this" or "render me that"
>>>>
>>>> For 1 and 2 to be relevant, messages would probably have to be sent by
>>>> the thousands or even millions; which is of course often the case, but
>>>> possible not in the context of the Command Pattern.
>>>>
>>>> So, in terms of human performance, I stand by my statement that working
>>>> with a structured message is slower, yet less prone to (again, human)
>>>> error.
>>>>
>>>> Thoughts?
>>>>
>>>> On Tuesday, May 20, 2014, Justin Israel <[email protected]> wrote:
>>>>
>>>>> I think you have made a completely inaccurate statement here. Or at
>>>>> least one that is ambiguous. You referred to a structured message as
>>>>> slower, and an unstructured message as faster because "it requires no
>>>>> parsing".
>>>>>
>>>>> When you send the json message across the wire, it will be a single
>>>>> message that will be read entirely and then it has to go through the json
>>>>> load process to be parsed and turned into an object on the other end. It
>>>>> also can only represent a few types in its standard spec.
>>>>>
>>>>> Protobuf has a binary format which can be smaller in size and faster
>>>>> to read because as it reads each index it can lookup the type from the
>>>>> proto and know what to read for the value.
>>>>>
>>>>> Have you done tests that suggest this fast/slow categorization is
>>>>> valid? Depending on your message size it could either be true, false, or
>>>>> negligible
>>>>> https://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking
>>>>>
>>>>> In that test, protobuf ended up being smaller and faster overall and
>>>>> was only slowest in the object creation component of the test.
>>>>>
>>>>> I'm just mentioning this because it immediately struck me as strange
>>>>> that you would represent the options by this criteria (fast/error prone,
>>>>> slow/less error prone). It usually depends on your message structure and
>>>>> size and required profiling.
>>>>> On May 21, 2014 3:59 AM, "Marcus Ottosson" <[email protected]>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> -
>>>>>>
>>>>>> Unstructured is fast, more prone to error
>>>>>> -
>>>>>>
>>>>>> Structured is slow, less prone to error
>>>>>>
>>>>>> I should say, “..prone to *human* error”. The computer doesn’t care
>>>>>> either way.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 20 May 2014 16:57, Marcus Ottosson <[email protected]> wrote:
>>>>>>
>>>>>> Yes you are probably right.
>>>>>>
>>>>>> This topic is about JSON versus Protobuf for transmitting commands
>>>>>> across processes; let’s refer to these as *unstructured* versus
>>>>>> *structured* respectively. To be explicit, and for those not
>>>>>> familiar with protobuf or schemas and what not, here’s what we’re
>>>>>> comparing:
>>>>>>
>>>>>> -
>>>>>>
>>>>>> Unstructured is fast, more prone to error
>>>>>> -
>>>>>>
>>>>>> Structured is slow, less prone to error
>>>>>>
>>>>>> Example
>>>>>>
>>>>>> *unstructured - client*
>>>>>>
>>>>>> >>> message = {
>>>>>> 'command': 'create',
>>>>>> 'args': ['age', '5'],
>>>>>> 'id': 'uuid' # Unique id for client
>>>>>> }>>> packed_message = json.dumps(message)>>>
>>>>>> socket.send(packed_message)>>> return_value = socket.recv()# Blocks
>>>>>> until server responds>>> # No post-processing required
>>>>>>
>>>>>> *unstructured - server*
>>>>>>
>>>>>> >>> packed_message = socket.recv()>>> message =
>>>>>> >>> json.loads(packed_message)>>> command = message['command']
>>>>>>
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBr0gSNLeDMk2uVZfK-s_c_g5w%3D88%3DU9u3BzmoA%3DNaLmw%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBr0gSNLeDMk2uVZfK-s_c_g5w%3D88%3DU9u3BzmoA%3DNaLmw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Python Programming for Autodesk Maya" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0pHAAgzw2zjN%2Bj0MQgsdMYh-cRL4rRQttWYLAu21OTWw%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0pHAAgzw2zjN%2Bj0MQgsdMYh-cRL4rRQttWYLAu21OTWw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>
>>>> --
>>>> *Marcus Ottosson*
>>>> [email protected]
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Python Programming for Autodesk Maya" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAMGtKXS6%3DJvKxJChsTSkBce1O6VkB5E_3eiYrtCm590Q%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAMGtKXS6%3DJvKxJChsTSkBce1O6VkB5E_3eiYrtCm590Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Python Programming for Autodesk Maya" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0FW-KWDM8wS2Cs70gp7mo7qxnCxk4qAzdG8hLHdd66Fw%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0FW-KWDM8wS2Cs70gp7mo7qxnCxk4qAzdG8hLHdd66Fw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> *Marcus Ottosson*
>> [email protected]
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Python Programming for Autodesk Maya" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBUotm6YH8OLa%2B7yqrmjO6s4JB2mpQ%3DduY2K0ZccuOZAA%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBUotm6YH8OLa%2B7yqrmjO6s4JB2mpQ%3DduY2K0ZccuOZAA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Python Programming for Autodesk Maya" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0%3Da6pYj6wbm47YKYaoaPjtnbyVCYuuo_O2LiLWrjxZYw%40mail.gmail.com<https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0%3Da6pYj6wbm47YKYaoaPjtnbyVCYuuo_O2LiLWrjxZYw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
*Marcus Ottosson*
[email protected]
--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOALctxwcSz1TdbJd87yNbhS2EH4iS3Db-xz2cwriCCJHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
